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PREFACE 


Two  questions  have  dominated  analysis  of  the  weapon  acquisition 
process  in  recent  years  and  are  especially  relevant  to  an  environment 
of  constrained  military  budgets:  What  is  an  appropriate  procurement 
strategy  for  a  particular  system?  How  can  we  increase  the  reliability 
of  cost-effective  comparisons  between  systems  with  uncertain  costs? 

This  Memorandum  provides  insights  into  each  question.  First,  it 
analyzes  the  closeness  of  actual  to  predicted  costs  for  weapon  systems 
of  the  1960s  and  compares  the  results  with  a  similar  analysis  of 
earlier  postwar  experience.  Second,  it  suggests  a  way  of  anticipating 
the  cost  outcomes  of  acquisition  programs  that  involve  various  levels 
of  technological  risk. 

The  research  reported  here  is  an  outgrowth  of  work  first  reported 
in  preliminary  form  in  RM-6072-PR,  System  Acquisition  Experience.^ 

The  model  described  in  Section  IV  of  the  Memorandum  has  been  refined, 
with  primary  emphasi 3  on  describing  the  underlying  motivation  for  the 
methodology  of  cost  factor  comparison 

This  Memorandum,  and  particularly  Sections  II  and  III,  is  some¬ 
what  more  technical  than  RM-6072-PR.  Specialists  who  are  concerned 
with  statistical  analysis  of  the  acquisition  process  are  the  main 
components  of  the  intended  audience  for  those  parts.  Sections  I  and 
IV  (Discussion)  and  the  hypothetical  example  of  a  predictive  applica¬ 
tion  of  the  model  (in  Section  V)  should  be  of  particular  interest  to 
decisionmakers  in  the  Deaprtment  of  Defense  and  to  the  acquisition- 
policy  echelons  of  the  Army,  Navy,  and  Air  Force.  For  them,  this 
study  may  offer  insight  into  the  probable  cost*  growth  and  range  of 
uncertainty  of  future  procurements  —  ijf  the  recent  past  is  a  suitable 
guide  to  such  insight. 

Empirical  results  presented  here  are  based  on  a  modified  version 
of  the  data  used  in  RM-6072-PR.  Two  features  of  this  data  base  should 

"hi.  L.  Perry,  D.  DiSalvo,  G.  R.  Hall,  A.  J.  Hannan,  G.  S.  Levenson, 
G.  K.  Smith,  J.  P.  Stucker,  System  Acquisition  Experience,  The  Rand 
Corporation,  RM-6072-PR,  November  1969. 
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be  kept  in  mind  in  evaluating  the  statistical  results.  First,  data 
were  available  for  only  a  small  number  of  systems.''"  The  data  for  the 
systems  of  the  1960s  were  originally  obtained  in  the  spring  of  1969 
by  the  Director  of  Defense  Research  and  Engineering,  aided  by  Rand, 

2 

through  questionnaires  on  21  systems  developed  by  the  three  services. 
That  survey  information  has  been  expanded,  refined,  and  up-dated  to 
December  1969  for  use  in  the  final  phase  of  estimation  of  this  analysis. 
Sufficient  information  for  the  model  has  now  been  obtained  for  15  pro¬ 
grams.  For  some  of  the  more  recent  of  these  programs,  projections  of 
"actual"  cost  or  time  of  operational  status  are  still  preliminary. 

Second,  the  measure  of  technological  advance  used  in  the  analysis 
was  obtained  by  a  rather  limited  survey  conducted  within  Rand.  These 
measures  are,  at  best,  crude  approximations  of  the  concept  of  the  rela¬ 
tive  level  of  technological  advance  sought  in  the  development  of  various 
weapons  over  two  decades.  Further  work  is  in  progress  to  refine  and 
measure  this  concept. 

■*■566  also  Perry  et  al .  ,  System  Acquisition  Experience,  especially 
Sections  I  and  II. 

2 

DDR&E  chose  the  sample  of  systems  with  the  advice  of  the  indi¬ 
vidual  services. 

3 

A  detailed  discussion  of  these  data  is  presented  in  the  Appendix. 
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SUMMARY 


'.’his  Memorandum  describes  a  methodology  for  comparing  and  predic¬ 
ting  c^st  experiences  of  weapon  system  procurements,  using  cost  factors 
(the  ratios  of  actual  to  predicted  costs)  as  a  basis.  Although  cost 
factors  can  be  greater  or  less  than  unity,  past  experience  has  shown 
that  in  the  usual  case  actual  costs  exceed  predicted  costs.  Cost 
factors  having  a  value  greater  than  unity  can  be  caused  by  inaccurate 
cost  estimates  (implying  that  estimates  are,  on  the  average,  lower 
than  efficiently  executed  actual  outcomes  would  be)  or  by  cost  growth 
arising  from  such  factors  as  scope  change  (alteration  of  goals  or  speci¬ 
fications  subsequent  to  the  start  of  a  development  program) ,  technolog¬ 
ical  uncertainty  (the  general  unpredictability  of  cost  for  risky  new 
technology),  or  various  inefficiencies.  Cost  factors  alone  do  not  pro¬ 
vide  sufficient  information  to  distinguish  among  such  contributors. 

Also,  cost  factors  are  not  adequate  by  themselves  for  making  cost 
comparisons,  since  different  types  of  systems  may  be  more  or  less  dif¬ 
ficult  to  estimate  accurately.  For  example,  one  weapon  system  may  have 
a  cost  factor  of  1.2  and  another  a  cost  factor  of  1.4.  But  if  the  first 
was  a  brief  program  involving  essentially  off-the-shelf  technology  and 
the  second  required  a  major  advance  in  technology  and  its  cost  had  to 
be  predicted  very  early  in  the  conceptualization  stage  of  the  program, 
the  apparent  40  percent  increase  in  costs  of  the  second  might  well  in 
retrospect  be  considered  to  be  money  well  spent  and  the  20  percent 
increase  of  the  first  could  be  considered  as  evidence  of  unacceptably 
poor  estimation  or  inefficient  management. 

An  attempt  to  make  adequate  cost  comparisons  must  take  into  account 
the  influence  on  costs  of  other  aspects  of  procurement,  including  pro¬ 
gram  duration  and  the  degree  of  technological  difficulty  encountered 
in  development.  This  proposition  leads  to  recognition  of  the  need  for 
a  theoretical  model  within  which  the  various  influences  on  costs  can 
be  structured. 

The  model  developed  here  considers  several  of  the  more  pertinent 
influences  on  the  size  of  a  cost  factor.  One  influence  is  program 
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length.  A  longer  program  may  result  in  a  high  cost  factor  either 
because  of  increased  time  during  which  the  management  of  the  program 
is  inefficient,  or  because  the  cost  estimate  was  based  on  a  very  early 
(and  therefore  vague)  configuration  of  the  weapon.  Another  influence 
on  the  cost  factor  is  the  requirement  for  advanced  performance  of  the 
system  leading  to  a  large  desired  technological  advance  and  therefore 
a  greater  intensity  of  effort  (for  a  given  program  length)  in  develop¬ 
ment  of  the  system.  This  may  result  in  a  high  cost  factor  because  of 
the  increased  probability  of  development  difficulties  or  because  of 
estimation  optimism  that  such  difficulties  will  not  occur. 

The  model  is  analyzed  using  two  sets  of  data  —  the  first  essen¬ 
tially  composed  of  systems  developed  in  the  1950s,  and  the  second  from 
the  1960s.'''  The  various  structures  for  the  model  all  indicate  that 
both  longer  programs  and  larger  technological  advances  usually  lead  to 
hij  er  cost  factors.  The  comparison  of  acquisition  experience  between 
the  two  decades  is  based  on  the  "Aircraft  and  Missiles"  subsample  of 
the  1960s  data,  which  is  most  closely  comparable  to  the  1950s  data. 

If  the  parameter  estimates  for  the  model  can  be  characterized  as  de¬ 
scriptive  of  the  system  acquisition  proceBB ,  then  the  statistical  results 
show  essentially  no  (net)  change  in  the  process  over  the  last  two  decades. 
That  is,  for  programs  comparable  in  length  and  difficulty,  1960s  pro¬ 
curements  would  have  resulted  in  actual  costs  exceeding  estimates  by 
roughly  the  same  proportion  as  had  1950s  procurements.  However,  there 
is  evidence  that  the  "typical"  program  of  the  last  decade  was  structured 
somewhat  differently  from  the  previous  experience.  On  the  average, 
development  program  length  seems  to  have  been  somewhat  shorter  and  the 


In  the  text,  quotation  marks  are  placed  around  these  decades 
when  they  are  used  to  name  the  samples,  which  cover  somewhat  broader 
ranges  of  years  (see  Section  III).  These  samples  were  compiled  at 
different  timeB  and  use  somewhat  different  bases  for  measurement  of 
cos ts  of  the  systems.  In  particular,  the  method  of  price  deflation 
and  assumptions  cf  quantity  adjustment  of  the  actual  and  estimated 
costs  are  different;  the  refinement  of  the  1960s  costs  were  chosen 
to  be  conservative  --  to  err,  if  at  all,  in  the  direction  of  under¬ 
stating  rather  than  overstating  any  improvement  in  cost  accuracy 
between  the  decades  (see  the  Appendix) . 
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magnitude  of  technological  advance  sought  has  been  kept  somewhat  lower 
than  In  the  1950s.  When  the  (different)  "typical"  programs  of  each 
decade  are  predicted  by  the  acquisition  process  (equation)  character¬ 
istic  of  the  decade,  the  cost  factor  for  the  1960s  Is  somewhat  lower. 
There  Is  thus  some  Indication  of  leas  bias  In  cost  estimation  or  over¬ 
run  of  actual  costs  in  the  1960s  than  previously . ^ 

Projections  can  also  be  made  concerning  the  probable  difference 
between  actual  and  predicted  costs  of  future  programs  as  well  as  a 
range  of  uncertainty  of  estimation  inaccuracy  or  actual  cost  growth  — 

provided  that  such  future  programs  are  sufficiently  similar  to  those 

2 

in  the  present  sample.  However,  the  data  that  produced  these  results 
are  crude.  Thus,  conclusions  concerning  the  range  of  future  uncer¬ 
tainty  should  be  considered  as  tentative  and  subject  to  refinement 
by  improved  measurement  of  the  appropriate  variables  and  modeling  of 
their  interrelationships. 

The  results  only  suggest  the  range  of  possible  cost  uncertainty 
based  on  past  experience;  the  policymaker  must  still  cope  with  the 
fact  that  weapon  systems  involving  requirements  for  high  levels  of 
technological  advance  will  continue  to  be  components  of  our  force 
structure.  Acquisition  of  such  systems  involves  unavoidable  cost 
uncertainty.  If  this  uncertainty  is  to  be  reduced  or  hedged,  there 
must  first  be  a  substantial  improvement  in  the  effectiveness  of  pro¬ 
curement  strategies  and  processes  —  which  may  require  fundamental 
changes  in  organization  and  decisionmaking  as  well  as  in  incentives 
and  procedures. 


See  Section  IV  for  a  full  elaboration  of  these  results;  the 
"Discussion"  subsection  reviews  the  equation  results  and  analyzes 
them  graphically. 

2 

And,  of  course,  provided  that  the  cost  estimates  have  been 
obtained  by  a  method  that  has  net  already  "corrected"  them  for  the 
"usual"  overall  growth  of  costs.  The  method  of  projection  is  dis¬ 
cussed  by  example  in  Section  V. 
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I.  INTRODUCTION 


A  complete  analysis  of  weapon  system  procurements  should  weigh 
the  costs  of  various  systems  against  their  effectiveness.  Effective¬ 
ness,  of  course,  ought  to  be  measured  in  terms  of  (1)  the  extent  to 
which  the  weapon  produced  meets  the  threat  actually  present  at  the 
time  when  the  weapon  is  in  the  inventory,  and  (2)  the  length  of  time 
during  which  it  remains  an  important  element  of  the  inventory.  Tra¬ 
ditionally,  however,  weapon  procurement  studies  have  had  a  more 
limited  goal. 

When  a  weapon  system  acquisition  is  reviewed  and  evaluated, 
whether  within  the  Department  of  Defense  or  by  Congress ,  the  closeness 
of  the  actual  cost  to  earlier  estimates  is  a  common  concern.  A  cost 
factor  —  the  ratio  of  the  actual  cost  to  the  estimated  cost  —  is 
frequently  used  as  a  measure  of  the  degree  of  closeness.  We  must 
have  legitimate  methods  of  using  popular  measures  such  as  cost  factors 
in  these  evaluations,  but  we  must  also  appreciate  the  more  limited 
role  of  cost  factors  within  a  complete  cost-effectiveness  analysis 
of  the  procurement  process.  As  an  illustration,  consider  the  condi¬ 
tions  necessary  for  cost  factors  to  be  of  central  importance  in  a 
co»t-effectivenesa  analysis. 

Suppose  that  a  new  weapon  system  is  contemplated  and  that  we  know 
with  certainty  that  it  will  perform  as  expected  and  be  available  to 
the  inventory  when  expected.  Let  us  also  suppose  that  all  other  weapon 
systems  being  considered  for  procurement  have  this  performance  and 
schedule  certainty.  We  would  then  know  the  effectiveness  of  each  of 
the  alternatives  only  to  the  extent  that  we  could  be  confident  of  our 
knowledge  of  the  presence  and  extent  of  the  threat  that  these  systems 
would  counteract.  Only  in  the  case  of  certain  knowledge  of  future 
threats  would  the  cost-effectiveness  comparison  of  the  weapon  systems 
reduce  strictly  to  the  cost  components.  Then  the  probability  of 
selecting  the  most  cost-effective  weapon  systems  would  be  directly 
related  to  the  accuracy  with  which  we  could  predict  the  costs. 
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The  accuracy  of  Che  cost  estimate  has  several  dimensions.  Two  of 
the  essential  features  are  bias  and  dispersion.  Bias  refers  merely  to 
any  consistent  tendency  for  estimates  to  be  lower  or  higher  than  actual 
cost  outcomes;  dispersion  relates  to  the  probability  that  an  individual 
estimate  may  differ  f.'om  the  actual  cost,  even  after  the  usual  bias 
of  estimates  is  taker.  Into  account.  Assume  that  two  weapon  systems, 
with  approximately  the  same  range  of  estimated  costs,  have  been  pro¬ 
posed  to  combat  a  particular  threat.  If  one  type  of  weapon  system 
tended  to  have  an  upward  bias  in  its  cost  estimates,  and  another  type 
had  downward-biased  estimates,,  the  cost-effective  comparison  between 
these  types  of  weapons  might  imply  the  wrong  selection.  There  would 
be  a  high  probability  of  selecting  the  second  type,  when  in  fact  the 
first  would  be  the  better  choice.  Similarly,  if  the  dispersion  of 
cost  estimates  of  weapon  systems  is  high,  we  should  be  less  confident 
of  correct  cost-effective  comparisons  between  contemplated  procurements 
because  we  would  be  less  confident  that  current  estimates  (on  which  a 
decision  must  be  based)  are  sufficiently  close  to  what  the  systems 
will  actually  cost. 

Cost  factors  are  mainly  useful  in  the  bias  aspect  of  the  accuracy 
of  our  knowledge  about  weapon  system  costs.  For  this  reason,  compari¬ 
son  of  cost  factors  is  a  sufficient  procedure  for  evaluating  procure¬ 
ment  experience  only  when  dispersion  as  well  as  the  other  possible 
uncertainties  mentioned  above  are  not  present.^ 

Aside  from  its  limits  in  importance  in  cost-effectiveness  studies, 
the  cost  factor  also  has  limited  interpretability .  For  example,  if 
a  weapon  system  is  estimated  to  cost  $4  million  and  actually  costs 
$5  million,  the  cost  factor  for  this  system  is  1.25.  Many  times  this 
is  described  as  "25  percent  cost  overrun."  But  the  "blame"  for  the 
discrepancy  between  estimate  and  actual  really  cannot  be  identified 
simply  from  their  ratio.  Inspection  of  the  value  of  a  cost  factor 
only  tells  the  degree  to  which  the  actual  cost  and  the  estimated  cost 

^"The  methodology  developed  here  allows  for  statistical  analysis 
within  which  dispersion  as  well  as  bias  can  be  investigated.  See 
Sections  IV  (Discussion)  and  V  below. 
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differ;  it  does  not  reveal  the  reason  for  the  difference.  The  cost 
factor  by  itself  does  not  distinguish  between  bias  and  overrun;  the 
estimate  may  have  been  poor  ojr  the  manufacturer  or  program  management 
may  have  been  inefficient. 

In  summary,  cost  factors  are  of  limited  value  to  the  analysis  of 
procurement  aa fecriveness  in  several  important  respects.  Only  with 
confidence  in  our  information  on  schedule,  performance,  and  future 
threat,  as  well  as  with  confidence  of  no  dispersion  component  to  our 
cost  estimating  accuracy  would  cost  factors  be  the  sole  determinant 
of  cost-effectiveness  comparisons.  Even  if  we  know  how  large  cost 
factors  will  be,  additional  information  is  needed  (about  estimating 
procedures,  or  contractor  or  management  methods,  incentives  and 
objectives)  before  we  can  determine  how  they  can  be  reduced.  Still, 
cost  accuracy  is  one  element  of  the  procurement  effectiveness  question. 
If  cost  accuracy  has  improved  over  past  experience,  we  can  be  more 
confident  than  in  the  past  of  the  correctness  of  our  decisions  based 
on  cost-effectiveness  comparisons . 

The  analysis  to  be  presented  in  the  following  pages  deals  essen¬ 
tially  with  how  one  might  compare  the  degree  of  cost  accuracy  between 
two  periods  of  time.  The  specific  application  will  be  a  comparison  of 
the  experience  in  cost  accuracy  in  the  1960s  with  the  experience  of 
the  1950s.  The  question  of  why  our  actual  experiences  differ  between 
the  two  decades  and  what  these  differences  may  be  attributed  to  is  not 
the  direct  objective  of  this  Memorandum.^  The  empirical  estimates  of 
the  mocel  can  also  be  used  to  predict  probable  cost  growth  and  range 
of  uncertainty  of  future  procurements  —  if  the  recent  past  is  a 
suitable  guide  for  such  insights. 

^See,  for  example,  the  research  reported  in  Perry  et  al. ,  System 
Acquisition  Experience,  and  a  proposal  for  future  improvement,  E.  Dews, 
luylng  (iompat+tivaly  After  Designing  to  a  Price,  The  Rand  Corporation, 
forthcoming. 
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PREVIOUS  METHODOLOGIES 

Although  studies  of  the  policy-related  issues  of  weapon  systems 
acquisition  have  varied  substantially,  the  actual  empirical  under¬ 
pinnings  of  these  analyses  have  been  quite  similar.^  Table  1  displays 
succinctly  and  somewhat  simplistically  the  evolution  of  studies  using 
cost  factors. 

Throughout  the  1950s  the  objective  was  to  find  the  earliest 
possible  estimate  for  a  given  project  and  divide  it  into  the  most 
accurate  and  up-to-date  appraisal  of  what  the  project  was  actually 
costing.  Cost  factors  during  this  period  were  the  ratio  of  unit  costs 
of  production  at  the  time  of  the  estimate  and  for  the  actual  production 
item.  The  quantity  proposed  at  the  time  of  estimate  was  often  sub¬ 
stantially  different  from  the  quantity  actually  produced.  Adjustment 
to  make  the  quantities  comparable  left  considerable  uncertainty  in 
what  the  cost  factor  for  a  particular  weapon  svstp™  <»hould  be.  The 
Marshall-Meckling  study  used  this  "earliest-estimate  basis"  and 
reported  averages  for  two  different  sets  of  cost  factors  for  the  same 
set  of  weapon  systems.  The  averages  reported  under  "B"  in  Table  1 
are  based  on  the  set  of  factors  calculated  by  Brussell  and  those 

under  "S"  are  based  on  the  set  derived  by  Sumners  for  the  same  weapon 

„  _  2 
systems . 


A.  W.  Marshall  and  W.  H.  Meckling,  Predictability  of  the  Costs, 
Time,  and  Success  of  Development,  The  Rand  Corporation,  P-1821,  December 
1959,  also  in  The  Rate  and  Direction  of  Inventive  Activity,  Princeton 
University  Press,  Princeton,  iS 62 ;  R.  Summers,  Cost  Estimates  as  Pre¬ 
dictors  of  Actual  Weapons  Costs:  A  Study  of  Major  Hardware  Articles, 

The  Rand  Corporation,  RM-3061-PR,  March  1965,  also  in  T.  Marschak, 

T.  K.  Glennan,  Jr.,  and  R.  Summers,  Strategy  for  R&D:  Studies  in  the 
Microeconomics  of  Development,  Springer-Verlag ,  New  York,  1967.  The 
extensive  data  base  compiled  by  E.  R.  Brussell  of  Rand  during  the  years 
1957-1961  formed  the  empirical  basis  for  these  two  studies.  See  also 
M.  J.  Peck  and  F.  M.  Scherer,  The  Weapon  Acquisition  Process:  An 
Economic  Analysis,  Harvard  University  Press,  Cambridge,  Massachusetts, 
1962;  and  F.  M.  Scherer,  The  Weapons  Acquisition  Process:  Economic 
Incentives ,  Harvard  University  Press,  Cambridge,  Massachusetts,  1964. 

^Marshall  and  Meckling ,  Predictability  of  the  Costs...,  pp.  13-15. 
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Tahle  1 
METHODOLOGIES 


MARSHALL  -  MECKLING  (1959) 

Method:  Means  of  Cost  1  actors  (Actual/uarliest  Estimate) 
Resuit:  Average  for: 

Fighters  Bombers  Cargo  &  Tankers  Missiles 

JL  _JL  JL  _JL  JL  JL  _B_  JL 

1.8  1.7  3.4  2.7  1.2  1.2  6.4  4.1 


Definitions : 

F  ■  cost  factor;  t  ■  the  timing  of  the  estimate  within 
the  development  program  (expressed  as  a  fraction  of  the 
program  length);  A  ■  level  of  technological  advance  sought; 

L  -  length  of  the  development  program;  and  T  ■  calendar  year. 
The  t  statistics  are  presented  in  parentheses  below  coeffi¬ 
cient  estimates. 
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The  Summers  study  contributed  further  to  the  methodological  basis 
for  evaluation  and  comparison  of  the  cost  aspects  of  weapon  acquisitions. 
Sumners  hypothesized  a  causal  relationship  between  the  bias  in  cost 
estimates  (as  reflected  in  part  by  the  e::tent  to  which  the  cost  factor 
deviates  from  unity)  and  certain  characteristics  of  the  project,  in¬ 
cluding  the  timing  of  the  estimate  within  the  course  of  the  project. 

In  other  words,  he  acknowledged  that  there  was  often  more  than  one 
estimate  of  what  the  project  would  cost  and  that  these  estimates  would 
tend  to  be  more  accurate  (or,  at  least,  closer  to  the  actual  outcome) 
at  later  stages  of  the  program.  By  taking  the  earliest  estimate  each 
time  and  comparing  them  across  projects,  an  investigator  could  not  be 
sure  of  having  found  an  equally  early  estimate  for  each  project  or 
even  equally  early  ones  within  types  of  projects.  Summers  therefore 
related  the  cost  factor  to  the  timing  of  the  estimate  within  the 
development  program  (t),  the  level  of  technological  advance  sought 
for  the  new  weapon  (A),^  the  length  of  the  development  program  (L) , 
and  the  year  in  which  the  estimate  was  made  (T) .  His  main  underlying 
hypothesis  was  that  as  the  proportion  of  time*,  increased,  the  estimators 
would  have  a  better  idea  of  what  the  program  would  actually  cost  and 
therefore  the  cost  factor  would  be  smaller. 

Summers  points  out  that  at  the  beginning  of  a  development  period, 
the  strategy  would  be  to  choose  from  among  the  various  possible  con¬ 
figurations  of  the  system  the  one  that  seems  to  be  the  least  costly 
to  achieve  the  performance  objectives.  As  time  goes  on,  the  trial 
and  error  method  of  achieving  the  desired  performance  leads  to  succes¬ 
sively  more  expensive  configurations.  Thus  the  cost  factor  will 
approach  unity  from  above  as  the  estimated  cost  approaches  the  actual 
cost  from  below.  He  al90  hypothesized  that,  other  things  equal,  a 
long  development  period  will  also  contribute  to  more  bias  in  the  cost 
factor.  Furthermore,  the  more  technologically  advanced  the  system 
under  development,  the  more  likely  it  is  that  those  less  expensive 
methods  that  have  some  hope  of  reaching  the  performance  goals  will 
not  in  fact  achieve  those  goals. 


This  survey  measure  is  also  used  in  the  analysis  reported  below. 
See  p.  25  for  a  description  of  A. 
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Finally ,  advancement  in  time  is  entered  simply  to  represent  the 
continuing  improvements  that  are  presumed  to  take  place  in  the  cost 
estimator's  ability  to  predict  the  actual  outcome  with  the  information 
available  to  him. 

The  form  of  Summers  '  model  is  simply  "the  most  satisfactory"  of 
a  variety  of  functional  forms  that  were  explored  empirically.  The 
indicated  directions  of  bias  from  the  features  elaborated  above  do, 
in  fact,  find  support  in  the  empirically  estimated  model  shown  in  Table 
1.  However,  the  estimated  values  for  the  part  of  the  model  quadratic 
in  A  lead  to  the  rather  peculiar  result  that  for  values  of  A  up  to 
about  11,  increasing  values  of  A  lead  to  less  bias,  whereas  for  values 
of  this  technological  advance  measure  above  that  point,  the  amount  of 
bias  increases.'*'  Although  the  time  variable  (T)  is  suggestive  of  the 
constant  progress  in  our  ability  to  estimate  what  costs  will  be,  it 
also  leads  r.o  the  rather  unfortunate  conclusion  that  by  the  1960s 
(earlier  or  later  in  the  decade,  depending  on  the  values  of  t,  A,  and 
L)  we  are  bound  to  be  predicting  costs  not  merely  less  optimistically 
(F  greater  than  but  closer  to  unity),  but  increasingly  pessimistically 
(F  less  than  unity  and  declining)  —  that  is,  increasingly  less 
accurately. 

It  is  vital  to  our  interpretation  of  such  results  that  we  under¬ 
stand  the  motivation  underlying  a  model  for  analyzing  cost  factors 

before  we  can  elaborate  an  explicit  formulation  amenable  to  empirical 
2 

analysis . 

THEORETICAL  MODEL 

To  understand  the  reasons  for  an  overrun  or  a  bias  to  be  reflected 
In  a  cost  factor,  one  must  characterize  the  way  actual  and  estimated 
costs  might  arise.  Costs  must  be  related  to  other  important  features 

^Summers'  sample  values  of  A  ranged  from  5  to  16. 

2 

The  discussion  of  the  theoretical  model  presented  in  this  next 
section  provides  the  basis  for  the  analysis  in  Section  IV  of  Perry  et  al. , 
System  Acquisition  Experience,  as  well  as  for  the  elaboration  of  that 
analysis  reported  in  this  Memoranda.  See  also  Peck  and  Scherer,  The 
Weapons  Acquisition  Process,  Part  III  (esp.  chaps.  9  and  11). 
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of  an  acquisition  program  —  schedule,  performance,  intensity  of  effort, 
and  technological  advance  sought.  This  characterization  is  not  an 
attempt  to  describe  mathematically  the  way  costs  actually  arise  from 
either  production  or  management  activities,  nor  is  the  estimated  cost 
function  an  attempt  at  mathematically  describing  the  cost  estimating 
procedures  that  have  been  used.  The  mathematical  relationships  are 
essentially  an  attempt  to  capture  the  major  influences  on  costs  with¬ 
out  trying  to  capture  the  mechanism  by  which  they  influence.  That 
is,  to  analyze  a  phenomenon  empirically,  it  is  not  necessary  to 
describe  in  detail  the  mechanism  by  which  the  phenomenon  occurs.^ 

The  notation  that  will  be  used  for  this  discussion  is  presented 
in  Table  2.  The  cost  aspect  of  the  model  can  be  characterized  by  two 
equations : 


C 

a 


fl<La’EVRa> 


(1) 


C 

e 


f9(L  |E  , R  ) 
z  e  e  e 


(2) 


The  main  emphasis  in  this  model  is  on  a  narrowing  in  the  range  of 

potential  outcomes  that  is  achieved  during  the  development  portion 

of  the  program.  Thus,  two  of  the  three  arguments  of  the  f^  actual 

cost  function  pertain  to  development  —  program  length  and  intensity 

of  effort.  The  third  argument  is  merely  the  resources  required  to 

2 

produce  the  system  configuration  settled  upon  in  development. 

It  is  not  unreasonable  to  assume  further  that  the  decisi  )ns  made 
during  development  determine  the  resources  needed  during  production; 


University  of  Chicago  Press,  Chicago,  1963,  Chapter  I. 

2 

A  production  function  characterization  (labor  and  capital  services 
as  well  as  "learning  by  doing"  resulting  in  units  of  the  system)  would 
be  a  reasonable  approach  to  achieving  a  measure  of  "resource  require¬ 
ments."  This  will  not  be  elaborated  here.  Instead,  it  is  merely 
necessary  to  assume  that  decisions  made  as  a  result  of  the  development 
phase  —  characterized  by  La  and  Ea  —  imply  a  certain  commitment  of 
resources  to  produce  the  system  specified. 
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Table  2 

NOTATION  FOR  CHARACTERIZING  A  PROGRAM 


Definitions^ 

A  Technological  advance  sought 
C*  Total  program  optimal  (that  is,  minimum)  cost 
Total  program  actual  cost 
Total  program  estimated  cost 

E*  Optimal  (that  is,  minimum-cost)  level  of  effort  (for  example, 
engineering  man-hours)  per  month  for  development 

Actual  level  of  effort  per  month  for  development 

E^  Estimated  level  of  effort  per  month  for  development 

L*  Optimal  (that  is,  minimum-cost)  development  program  length,  in 
months 

Lg  Actual  months  for  development 
Lg  Estimated  months  for  development 
P  Desired  performance 

Rg  Actual  resources  devoted  to  production  of  the  system 
R£  Estimated  resources  devoted  to  production 
Identity 

C 

Cost  Factor:  F  ■  — — 

e 

Note: 

■^The  variable  "M",  which  is  used  as  a  surrogate  for  program  length 
throughout  the  empirical  analysis,  is  defined  in  Section  III. 
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that  Is , 


and 


R 

a 


£3(La'Ea>- 


R 

e 


vw* 


(3) 

(A) 


For  example,  if  an  aircraft  is  to  be  designed  during  a  short  development 
phase  with  minimum  effort,  the  designers  may  simply  choose  to  use  a 
costly  to  produce  but  powerful  fan  jet  engine;  if  they  have  more  ti:>as 
for  designing,  they  may  be  able  to  achieve  desired  performance  objectives 
with  a  turboprop  engine.  These  decisions  determine  the  cost  of  the 
production  phase  of  the  procurement. 

The  usual  U-shaped  curves  relating  costs  to  program  length  are 
shown  in  Fig.  1.  There  is  presumed  to  be  a  development  strategy  — 
program  length  and  development  effort  —  that  will  lead  to  the  minimum 
total  cost  (C*)  of  development  and  production  (see  Fig.  la).  For 
shorter  program  lengths,  lower  development  costs  might  arise,  perhaps 
by  choosing  at  the  outset  a  quite  costly  configuration  to  produce  but 
one  that  is  reasonably  certain  to  achieve  performance  goals.  Thus, 
the  total  costs  —  including  the  high  production  costs  —  will  be 
larger  than  C*.  This  figure  can  be  '•.bought  of  as  a  range  of  procure¬ 
ment  strai agies  from  concurrency  of  development  (shorter  development 
programs  in  which  all  components  are  developed  concurrently  with  inte¬ 
gration  of  the  system  at  the  last  stage  of  development)  to  sequential 
development  (longer  programs  in  which  the  essential  parts  of  the  con¬ 
figuration  are  assembled  and  tested  earlier  in  the  development  phase)  . 

The  curve  of  Fig.  la  indicates  that  a  very  compressed  schedule  for 
development  may  be  able  to  achieve  the  desired  system  (characterized, 
perhaps,  by  an  elaborate  set  of  minimum  performance  goals),  but  that 
one  can  also  seek  to  design  a  minimum  cost  development  strategy.'*' 

For  a  different  weapon  with  more  demanding  performance  requirements 

^The  minimum  of  the  curve  could  actually  remain  flat  over  a  range 
of  development  program  lengths. 
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C* 


L 


(b)  Estimated  costs 


Fig.  1— Relationship  of  cost  to  program  length 
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the  requisite  levels  of  effort,  program  length,  or  resources  for  pro¬ 
duction  would  shift  upward. 

In  simmary,  equation  (1)  deals  with  features  affecting  the  actual 
program  costs,  which  are  presumed  to  be  related  to  the  actual  develop¬ 
ment  schedule  (L  )  ,  level  of  effort  (E  ) ,  and  requisite  production 

di  3l 

resources  (R  ) .  Note  that  the  degree  of  cost  increase  from  a  schedule 

di 

stretchout,  for  example,  is  not  necessarily  assumed  to  be  equal  to  the 
cost  increase  associated  with  an  equal  amount  of  schedule  compression 
(that  is,  the  "U"  shaped  curve  in  Fig.  la  is  probably  not  symmetric). 

In  fact,  the  curve  undoubtedly  approaches  an  asymptote  at  least  for 
very  short  lengths  of  development.  A  final  influence  on  actual  costs 
is  the  total  accumulation  of  inefficient  contractor  performance  or 
administrative  laxity  in  controlling  costs.  This  phenomenon  may  be 
captured  in  large  part  by  the  length  of  time  of  the  development  program 
(L  )  and  the  resources  in  production  (R  )  .  ^ 

cl  3 

Cost  estimates  characterized  by  equation  (2)  take  into  account 

2 

the  same  basic  influences,  except  that  there  is  an  added  interpre¬ 
tation  that  can  be  given  to  the  inclusion  of  L^.  This  term  not  only 
reflects  the  "U"  shaped  curve  of  higher  costs  for  deviations  from 
optimal  program  length,  but  may  also  capture  a  range  of  behavior  from 
accurately  estimating  the  inefficiencies  in  the  procurement  procedures 
to  estimating  costs  in  an  increasingly  optimistic  or  myopic  manner  as 
the  distance  to  the  end  of  development  of  the  system  increases. 

These  equations  form  a  scheme  for  translating  reasonable  hypo¬ 
theses  by  which  costs  and  cost  estimates  arise  into  a  scheme  for 
analyzing  the  bias  in  cost  estimates  or  the  extent  of  cost  overruns, 
using  cost  factors.  To  see  this  translation,  consider  a  cost  factor, 
which  is  a  measure  of  this  bias  or  overrun: 

■^In  this  use,  may  be  somewhat  deficient  indicator  of  ineffi¬ 
cient  use  of  resources  in  production  [through  equation  (3)]. 

2 

Let  me  emphasize  again  that  I  am  not  trying  to  suggest  a  method 
for  actually  estimating  costs,  but  only  for  summarizing  the  process 
sufficiently  accurately  that  it  can  be  analyzed  empirically. 
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F-r-  ■  <5) 

e 

Substituting  from  equations  (1)  to  (4)  above  into  (5)  yields 

F  -  F(LLEE)  (6) 

a  e  a  e 

F  deviates  from  unity  (perfect  estimate  and  perfect  cost  control  or 
the  same  extent  of  overestimate  and  cost  slippage)  only  through  the 
following  mechanisms : ^ 

o  The  height  of  the  minimum  cost  level  is  underestimated. 

o  The  effect  on  costs  of  deviations  from  optimal  develop¬ 
ment  program  length  is  underestimated. 

o  The  effect  on  costs  of  deviations  from  optimal  develop¬ 
ment  effort  is  underestimated. 

o  Management  or  production  inefficiency  is  not  accounted 
for  in  the  estimate,  or  is  misjudged. 

The  first  point  is  simply  the  possibility  of  direct  bias  due  to  the 
cos t  estimator's  understanding  of  the  height  of  the  usual  schedule- 
to-cost  curve  (Fig.  lb).  It  may  be,  for  example,  that  he  thinks  the 
curve  (and  therefore  the  minimum  point)  is  lower  than  it  actually  is, 
and  therefore  the  line  at  the  level  of  actual  minimum  costs  (C*) 
intersects  the  estimated  curve,  rather  than  being  tangent  to  the 
curve  as  it  is  in  fact. 

Bias  may  also  be  due  to  the  cost  estimator's  optimism  in  the 
degree  to  which  schedule  compression  or  stretchout  leads  to  larger 
costs.  This  part  of  the  bias,  in  other  words,  comes  from  an  opti¬ 
mistic  view  of  the  steepness  of  the  sides  of  the  schedule-to-cost 
curve  (Fig.  la).  Another  possible  source  of  bias  comes  from  a  similar 
optimism  with  respect  to  the  effect  of  effort  on  cost.  The  final 
possible  source  of  bias,  as  well  as  the  cost  overrun  possibility, 


Bias  in  costs  has  almost  always  appeared  as  overly  optimistic 
estimates  or  actual  overruns  —  that  is,  cost  factors  greater  than 
unity.  Thus,  all  bias  influences  discussed  here  are  stated  in  those 
terms . 
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is  suggested  by  the  last  point  of  the  list.  There  may  be  inefficiency 
of  the  program  manager  or  contractor  in  conducting  the  program,  or 
uncertainty  within  which  the  cost  estimator  must  make  his  projection. 
These  possible  sources  of  bias  or  overrun  in  costs  from  length  of 
development  or  level  of  effort  are  summarized  graphically  in  Figs. 

2  and  3  (in  each  figure,  the  other  variable  is  held  constant). 

To  see  the  underlying  influences  behind  these  sources  of  bias 
in  costs,  we  must  look  further  into  the  other  two  features  of  a  pro¬ 
gram  that  are  co-determined  with  the  cost  of  the  acquisition;  namely, 
the  length  of  the  program  and  the  amount  of  effort. ^ 

The  measure  of  effort  per  unit  of  time  is  presumed  to  be  related 
to  schedule  and  performance  features  of  the  desired  system  according 
to  the  following  relationship : 

Ea  =  f5(La’A(P))’  (7) 

Ee  =  f  6  (l*e ,  A(P )  )  .  (8) 

There  is  an  important  assumption  embodied  in  equation  (7) ,  as  illus¬ 
trated  by  Fig.  4.  The  parable  for  following  Fig.  4  may  be  described 
as  follows:  The  conceptualization  of  a  new  weapon  system,  deemed 
essential  to  future  defense  against  a  perceived  threat,  takes  the 
form  of  specification  of  the  increased  performance  characteristics 
needed  from  the  new  weapon  (P).  These  characteristics  are  portrayed 
schematically  simply  as  the  height  up  the  vertical  scale  in  Fig.  4(a) . 
Corresponding  to  this  performance  characterization,  there  is  a  set  of 
technological  advance  levels  which,  if  attained,  make  it  likely  that 
the  performance  characterizations  specified  will  in  fact  be  achieved. 

The  very  fact  that  a  new  weapon  is  deemed  necessary  implies  that  the 
performance  characterization  of  this  new  weapon  will  be  somewhat  higher 
than  those  in  existing  stock,  and  therefore  that  the  level  of  P  will 

^Throughout  this  discussion,  effort  is  thought  to  be  measured  in 
terms  of  some  work  load;  for  example,  it  might  be  measured  as  engineering 
man-hours  per  month  for  the  development  phase  of  the  acquisition. 
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Fig.  2— Relationship  of  cost  factor  to  program  length 


Fig. 3 — Relationship  of  cost  factor  to  effort 
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(b)  Relationship  of  effort  to  technological  advance  sought  for  a  given  program  schedule 


Fig. 4— Influence  of  performance  specification 
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never  be  zero.  Correspondingly,  the  fact  that  this  weapon  is  some¬ 
thing  new  implies  that  there  will  be  a  certain  minimum  technological 
advance  (possibly  measured  as  the  amount  of  skill  necessary  to  put 
together  a  new  configuration  based  on  off-the-shelf  components). 

The  parable  continues  to  the  assumption  that  within  this  band  of 
technological  advance,  a  particular  level  of  technological  advance 
is  sought.  Corresponding  to  this  level  and  corresponding  to  a  pre¬ 
scribed  schedule  for  achievement  of  the  new  weapon  system  (given  tech¬ 
nical  efficiency),  a  level  of  effort  is  uniquely  determined  [see  part 
(b) ,  Fig.  4],  This  is,  of  course,  a  highly  simplified  characterization 
of  the  way  in  which  weapon  procurements  are  actually  structured.  It 
is,  however,  a  useful  simplification  for  the  purpose  of  concentrating 
on  cost  aspects  of  weapon  systems,  while  still  taking  into  account, 
at  least  broadly,  the  performs  ice  features  of  the  systems. 

If  we  suppose  that  the  level  of  A  in  Fig.  4  corresponds  to  curve 
A^  in  Fig.  5,  then  the  increasing  values  of  L,  labeled  L^,...  in 
Fig.  4,  correspond  to  increases  in  L  on  the  horizontal  axis  of  Fig.  5. 
This  latter  figure  portrays  another  characteristic  of  equation  (7). 

In  addition  to  showing  how  performance  and  technology  influence  the 
level  of  effort  of  the.  development  phase,  it  portrays  the  assumption 
that  one  can  reduce  the  le 'el  of  effort  by  lengthening  that  program's 
development  (that  is,  the  length  of  time  until  the  weapon  is  to  become 
operational).  It  is  also  assumed,  however,  that  one  can  reduce  that 
effort  to  a  minimum  level,  at  which  point  any  increases  in  the  length 
of  the  program  incur  diseconomies.  For  technological  advance  level 
A,  that  point  in  length  of  program  corresponding  to  minimum  effort  is 
L3  (L-jS  in  Fig.  4  and  Fig.  5  are  the  same). 

■*"Ihi8  description  is  possible  only  for  performance  measured  in 
one  dimension  as  is  done  in  this  example.  The  appropriate  concept 
for  a  vector  of  performance  measures,  some  of  which  may  not  be  as 
advanced  as  for  existing  weapon  systems,  might  be  a  measure  of  the 
length  of  the  vector  relative  to  an  appropriately  selected  set  of 
basis  vectors. 
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Fig.  5 —Tradeoff  between  length  of  development 
and  intensity  of  effort 
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Equations  (9)  and  (10),  then,-  complete  the  model  by  specifying 
values  for  the  actual  program  length  and  the  estimate  for  that  length: 

La*f7(L*),  (9) 

Le=fg(L*).  (10) 

The  schedule  for  the  program  is  presumed  to  be  designed  with  cost  in 
mind  and  is  therefore  related  to  the  minimum-cost  schedule;  it  is  also 
influenced  by  purely  external  factors  such  as  the  desire  that  the 
system  be  available  prior  to  the  realization  of  the  anticipated  threat. 
Although  the  estimate  of  program  length  parallels  this  reasoning,  it 
may  be  overly  optimistic  of  the  minimum-cost  schedule  for  longer  pro- 

?  f 

grams  [that  is,  t-j  >  fg] . 

The  three  main  features  of  this  model  —  cost  estimation  bias 
or  cost  overrun,  length  of  development,  and  level  of  effort  —  can 
now  be  combined  graphically  as  shown  in  Fig.  6.  This  3-dimensional 
diagram  combines  the  descriptions  of  Figs.  2,  3,  and  5.  The  faint 
curved  lines  represent  a  surface  of  values  of  F  for  various  combina¬ 
tions  of  E  and  L.  This  surface  comes  into  the  "room"  delineated  by 
the  F-L  plane  (from  Fig.  2)  as  the  "back  wall";  the  F-E  plane  (from 
Fig.  3)  as  the  "side  wall";  and  the  E-L  plane  (from  Fig.  5)  as  the 
"floor."  On  the  "floor,"  the  minimum  E-L  combinations  for  a  given 
level  of  technological  advance  are  displayed.  For  each  technological 
advance  level,  the  minimum-cost  L  and  E,  (that  ia ,  L*  and  E*)  determine 
a  point  on  the  surface;  these  points  for  varying  levels  of  technologi¬ 
cal  advance  form  the  line  depicted  by  the  broken  arrow.  The  value  of 
any  F  on  the  surface  can  be  derived  from  equation  (5) . 

To  this  point,  we  have  not  even  considered  the  possibility  that 
the  level  of  technological  advance  sought  does  not  lead  to  achievement 
of  the  performance  characteristics  chosen  (recall  that  Fig.  4a  depicted 
a  band  of  uncertainty).  If  it  does  not,  or  if  the  performance  demands 
are  increased,  there  may  be  a  decision  in  the  course  of  the  program 
to  increase  A.  Thus,  in  Fig.  6,  the  arrcv  on  the  "floor,"  depicting 
an  outward  shift  in  the  possible  effort  and  program  length  combinations, 


/ 


Fig. 6— Combined  theoretical  description  of  cost  factors 
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can  be  considered  a  characterization  of  technological  uncertainty  or 
of  what  is  often  called  "changes  in  scope."  This  latter  expression 
is  usually  used  to  describe  either  the  customer's  or  the  developer's 
changing  the  objectives  of  a  program  during  its  development  phase. 

The  effect,  whether  large  or  small,  is  to  alter  the  configuration  or 
performance  of  the  article  being  designed  or  tested  and  thus  to  change 
the  optimal  program  length  or  intensity  of  effort.  It  is  a  reasonable 
assumption  that  the  development  program  is  even  less  "optimally"  struc¬ 
tured  to  achieve  the  modified  system  with  minimum  cost  than  it  was  to 
achieve  the  original  configuration.  Attempts  are  usually  made  to 
capture  the  tendency  for  scope  change  in  the  cost  estimate  based  on 
past  experience.  Nevertheless,  scope  change  almost  always  appears  aa 
an  addition  to  estimated  program  costs  and  is  an  important  contributor 
to  cost  growth. 

In  summary,  this  model  presents  a  simplified  representation  of 
the  influences  of  performance,  technological  advance,  and  scheduling 
on  the  degree  of  bias  or  overrun  in  coBts.  The  scheme  relates  the 
three  dimensions  of  cost  bias,  program  length,  and  intensity  of  effort. 
The  next  section  contains  an  explicit  model  specification  that  can  be 
analyzed  empirically. 
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III.  EMPIRICAL  FORMULATION 


Comparison  of  bias  in  cost  perf o’T.iance  between  the  1950s  and  the 
1960s  can  be  accomplished  through  the  model  presented  graphically  in 
Fig.  6.  This  comparison  can  be  made  by  measuring  the  extent  to  which 
the  curved  surface  characteristic  of  cost  factors  in  the  1950s  flat¬ 
tened  out  closer  to  the  unity  plane  parallel  to  the  "floor"  for  the 
1960s.1 

Equation  (6)  can  be  simplified  for  the  empirical  exploration 
because  the  acquisitions  under  investigation  have  all  either  been 
concluded  or  are  sufficiently  advanced  that  we  can  be  reasonably  con¬ 
fident  of  knowing  the  actual  cost  and  consistent  date  for  the  "end  of 

2 

development"  of  these  projects.  Thus  the  L^  argument  of  the  function 
in  equation  (6)  does  not  actually  need  to  enter  a  function  used  to 
analyze  the  data  available  to  us.  Although  this  is  true  for  analyzing 
the  historical  data,  in  using  any  such  analysis  to  predict  future 
results,  we  must  keep  in  mind  that  we  will  not  know  program  length 
with  accuracy,  just  as  we  will  not  know  the  cost  of  the  program  with 
accuracy  (see  Section  V) . 


In  evaluating  the  accuracy  of  estimates  of  weapon  system  costs, 
dispersion  around  the  actual  cost,  as  well  as  bias,  must  be  considered. 
As  mentioned  above,  we  cannot  be  very  confident  in  making  decisions 
concerning  choice  of  acquisitions  if,  on  the  average,  our  predictions 
are  correct,  but  for  any  particular  prediction,  the  actual  cost  could 
easily  differ  by  a  factor  of  two  or  three. 

The  model  used  in  this  analysis  is  directed  at  identifying  bias; 
other  aspects  of  the  total  question  of  predictive  accuracy  will  be 
considered  in  evaluating  the  empirical  results  in  Sections  IV  (Discus¬ 
sion)  and  V. 

2 

For  several  of  the  systems  included  in  the  DDR&E  survey  reported 
in  Perry  et  al .  ,  System  Acquisition  Experience,  the  program  was  not 
sufficiently  advanced  to  determine  the  date  with  the  accuracy  implied 
by  this  statement.  These  projects  were  therefore  excluded  from  the 
sample  and  are  not  included  in  the  list  of  available  projects  presented 
in  the  Appendix. 

To  simplify  notation,  La  is  replaced  by  L  in  this  section  and  sub¬ 
sequently.  Since  no  data  were  available  to  distinguish  Ea  from  Ee, 
a  general  intensity  of  effort  (E)  measure  will  be  used  in  the  empirical 
model . 
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The  natural  assumption  for  the  way  development  program  length 
(L)  and  intensity  of  development  effort  per  month  (E)  influence  the 
functions  specified  in  Section  II  is  that  their  product  (L*E)  is  the 
essential  feature  —  measuring  total  effort  in  development.  The 
following  equation,  then,  captures  the  general  nature  of  the  pre¬ 
diction  of  system  cost  bias  described  in  Fig.  6: 


F  ■  ae 


bL-E 


(ID 


As  mentioned  above,  it  is  assumed  in  using  F  in  this  equation  that 
the  procurement  is  sufficiently  far  in  the  past  that  actual  coat  is 
known.  The  cost  factor  is  presumed  to  be  exponentially  relatec,  to 
the  total  effort  involved  in  bringing  the  system  to  fruition.'*'  This 
formulation  captures  the  effect  on  cost  factors  characterized  in  the 
theoretical  model;  namely,  that  for  longer  programs  or  programs  re¬ 
quiring  high  levels  of  effort)  ,  the  bias  due  to  optimism  in  cost 
estimation  or  to  inefficiencies  in  program  management  tends  to  be 
larger.  Thus  the  coefficient  b  is  hypothesized  to  be  positive. 


Throughout  this  Memorandum  I  have  emphasized  that  the  basic 
motivation  for  this  analysis  is  to  develop  a  framework  within  which 
cost  factors  can  be  analyzed.  The  implication  from  such  an  empirical 
formulation  (by  multiplying  by  Cg)  is  that 


C  -  aC  e 
a  e 


bLE 


and  therefore  that 
3C 


a  urr  bLE 

— —  -  abEC  e 
3L  e 


-  bEC 

a 

That  is,  the  marginal  impact  on  actual  costs  of  a  change  in  development 
program  length  depends  not  only  on  the  monthly  effort  in  development 
but  also  on  the  magnitude  of  costs.  Thus,  implicit  in  the  use  of  a 
cost  factor  in  this  type  of  model  is  the  assumption  that  the  impact  of 
changes  on  cost  growth  is  in  percentage  terms  in  the  program's  costs 
and  not  simply  in  additional  dollars.  The  validity  of  tula  assumption 
is  conjectural;  further  investigation  is  continuing  to  determine  if 
another  structure  using  cost  factors  or  a  model  that  abandons  the  cost 
factors  of  common  reference  would  be  preferable. 
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Along  with  the  presumption  that  estimates  for  longer  programs 
will  tend  to  be  more  biased,  there  is  also  the  likelihood  that  these 
longer  programs,  or  ones  involving  larger  technological  advance,  tend 
to  have  generally  more  cost  uncertainty.  Thus,  as  the  total  program 
effort  (L*E)  Increases,  both  larger  bias  and  dispersion  of  the  esti¬ 
mates  are  presumed  to  occur.  The  exponential  form  of  the  function 
is  chosen  so  that  one  obtains  the  positive  association  between  bias 
and  intensity  of  program  effort.  Furthermore,  in  the  form  in  which 
the  equation  will  be  estimated,  heteroscedasticity  is  reduced.  Thus, 
the  equation  can  be  transformed  into  an  expression  linear  in  the  coef¬ 
ficients  as  follows : 

log  F  ■  log  a  +  bL*E  +  c  (12) 

in  which  e  is  the  statistical  disturbance  term.  The  argument  is  that 
the  variability  of  F  is  likely  to  increase  as  L*E  increases.  However, 
it  is  a  more  reasonable  assumption  that  the  variability  of  log  F 
remains  essentially  constant  (as  total  program  effort  Increases)  and 
thus  the  disturbance  in  equation  (12)  is  more  nearly  homos cedas tic. * 

The  final  requirement  for  analyzing  this  model  is  the  choice  of 
measures  for  the  development  program  characteristics,  intensity  of 
effort  (E)  and  program  length  (L) . 

The  data  available  for  this  study  are  presented  and  discussed 
in  the  first  section  of  the  Appendix.  The  two  sets  of  data  used  are 
called  "1950s"  and  "1960s"  samples  since  the  weapon  systems  in  each 
are  predominantly  from  these  periods.  However,  the  actual  range  of 
sample  observations  (from  earliest  estimate  to  latest  operational 


Of  course,  heteroscedasticity  could  be  reduced  in  other  forms  of 
the  model  besides  the  exponential  form.  And  in  fact,  the  empirical 
results  might  display  an  essentially  linear  slope  over  the  range  of 
estimates.  The  hypothesis  here  is  that  estimates  deviate  from  actual 
costs  at  least  by  a  linear  function  of  total  program  effort,  and  very 
possibly  at  an  increasing  rate.  Since  the  exponential  form  leads  to 
the  implication  of  very  large  increases  in  F  for  very  long  or  diffi¬ 
cult  development  efforts  (for  example,  as  L*E  increases),  one  should 
be  cautious  not  to  draw  inferences  about  systems  in  the  future  (as  in 
Section  V)  that  are  outside  the  range  of  observations  of  L*E  in  the 
current  samples. 
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status)  of  each  is  February  1947  .to  July  1961  for  the  "1950a,"  and 
January  1958  to  December  1969  for  the  "1960s."  Therefore,  quotation 
marks  will  be  used  on  decade  references  whenever  they  are  employed 
as  the  names  of  the  samples . ^ 

No  direct  measure  was  available  of  the  intensity  of  effort  in 
development  of  the  systems  in  these  samples.  However,  a  variable 
quantifying  the  magnitude  of  technological  advance  sought  for  each 
of  the  systems  under  consideration  has  been  obtained.  The  values  of 
this  variable,  A,  for  the  systems  of  the  "1950s"  were  compiled  through 
a  survey  of  Rand  engineers  taken  by  Robert  Summers.  The  respondents 
were  asked  "to  rate  subjectively  the  magnitude  of  the  improvement  in 

2 

the  state  of  the  art  required  for  each  of  the  development  programs." 

The  results  were  then  consolidated  to  yield  a  value  of  A  for  each 
system.  These  measures  wei'3  taken  as  given  for  use  in  this  study; 
no  adjustments  or  reevaluations  were  attempted.  Values  of  A  for  the 
"1960s"  systems  were  also  obtained  through  a  survey.  The  respondents 
were  asked  to  estimate  the  magnitude  of  technological  advance  sought 
on  a  scalra  of  0  to  20  (or  more)  and  were  Instructed  to  make  these 
estimates  comparable  to  the  measurements  for  the  "1950s."  The  esti¬ 
mates  were  then  averaged  to  obtain  a  value  of  A  for  each  system. 

Tiese  surveys  were  limited  in  scope  and  resulted  in  only  a  few  responses. 
Measurement  of  the  technological  advance  sought  is  thus  at  a  preliminary 
stage;  an  expanded  and  refined  survey  would  be  desirable.  This  survey 
measure  of  technological  advance  sought,  along  with  development 

■*The  data  used  to  establish  cost  factors  for  the  "1960s"  were  ori¬ 
ginally  gathered  by  Rand  during  a  study  for  DDR&E,  but  have  been  updated 
to  reflect  the  best  estimates  of  actual  costs  as  of  December  1969  for 
programs  still  in  progress. 

Factors  for  the  "1950s"  are  all  for  Air  Force  projects.  These  data 
were  gathered  by  Eugene  R.  Brussell  in  the  late  1950s  and  early  1960s, 
and  the  factors  were  calculated  by  Robert  Summers  for  use  in  Coat  Esti¬ 
mates  as  Predictors. . . .  The  cost  factors  themselves  have  been  adjusted 
for  both  quantity  and  price  level  changes  during  the  course  of  the  pro¬ 
grams.  See  Perry  et  al . ,  System  Acquisition  Experience,  Sections  I  and 
II  for  a  discussion  of  the  scope  and  limitations  of  the  original  "1960s" 
sample.  See  also  the  discussion  in  the  Appendix. 

2 

Summers,  Cost  Estimates  as  Predictors...,  p.  25. 
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program  length,  can  be  transformed  into  a  surrogate  for  the  intensity 
of  effort  variable  [see  equation  (7)].  Several  transformations  of  A 
(and  L)  into  E  will  be  investigated  in  the  empirical  results  reported 
in  Section  IV. 

To  understand  the  measure  of  development  program  length  used  in 
the  analysis,  consider  Fig.  7  which  displays  the  level  of  program 
activity  through  time.  "Program  activity"  might  be  defined  as  the 
amount  of  both  development  and  production  resources  committed  to  the 
program;  it  could,  perhaps,  be  measured  by  dollars  spent  per  year  or 
by  engineering  and  production  man-hours  per  year.  Figure  7  is  not 
intended  to  portray  the  course  of  any  particular  program  or  programs; 
it  is,  rather,  a  theoretical  representation  of  the  course  of  programs 
in  general.  Individual  programs  might  differ  in  several  important 
respects . 

First,  the  areas  beneath  the  development  and  production  curves 
would  probably  vary  with  the  magnitude  of  technological  advance  sought 
for  a  specific  program.  For  example,  the  area  beneath  the  development 
curve  might  be  larger,  relative  to  the  area  beneath  the  production 
curve,  for  higher  levels  of  technological  advance.  In  addition,  "humps" 
in  the  development  curve  might  arise  because  of  unsuccessful  tries, 
and  "humps"  in  the  early  part  of  production  might  occur  because  of  re¬ 
tooling  needs.  There  might  also  be  idditional  large  development  com¬ 
mitments  after  initial  operational  delivery  (IOD)  because  of  new  versions 
or  models  of  the  weapon  system.  Finally,  the  later  part  of  the  pro¬ 
duction  curve  might  vary  with  changes  in  the  total  number  procured 
and  the  timing  of  delivery,  for  such  purposes  as  unforeseen  replace¬ 
ment  needs  or  maintenance  of  the  production  line.  This  variability 
is  depicted  in  the  later  part  of  production  by  a  band  rather  than  a 
line.  Since  the  purpose  of  Fig.  7  is  to  present  a  simplified  and 
generalized  concept  of  the  course  of  program  activity  through  time, 
the  qualifications  listed  above  are  not  explicitly  depicted. 

Within  the  framework,  the  measure  of  L  for  the  systems  of  the 
1960s  is  reasonably  clear.  The  date  of  the  estimate  made  near  the 
time  of  DoD  approval  of  a  development  program  provides  an  initial 
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Fig. 7— Levels  of  program  activity 
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base  point  because  it  marks  the  approval  of  the  commencement  of  engi¬ 
neering  development  and  provides  the  least  ambiguous  indicator  of  the 
beginning  of  a  program.  IOD  was  chosen  as  the  end  point  because  it  seems 
reasonable  to  assume  that  most  of  the  development  uncertainties  captured 
by  the  model  are  resolved  by  the  time  of  IOD.'*’  The  justification  of 
this  end  point  follows  from  the  conceptualization  of  the  course  of  pro¬ 
gram  activity  as  displayed  in  Fig.  7.  IOD  occurs  after  the  development 
phase  is  largely  completed.  This  means  that  the  length  of  time  from 
program  approval  *  )  IOD  is  the  relevant  time  period  during  which  most; 
of  the  technological  uncertainty  leading  to  cost  uncertainty  in  de¬ 
velopment  is  resolved.  Also,  any  initial  difficulties  in  tooling-up 
will  be  overcome  during  this  time  so  that  the  size  of  this  investment 
would  also  be  knowr .  Thus,  two  estimation  uncertainties  captured  by 
the  model  (namely,  early  stage  of  conceptualization  and  major  tech¬ 
nological  advance)  and  a  major  production  uncertainty  (tooling-up 

2 

cost)  will  be  largely  resolved  by  7.0D,  If  the  contractor  is  not 


This  end  point  was  chosen  mainly  because  such  information  was 
available  for  all  systems  in  this  decade.  Initial  operational  capa¬ 
bility  (IOC)  would  be  another  possible  choice  for  the  end  of  the  pro¬ 
gram;  however,  the  data  available  were  not  sufficient  to  allow  the 
use  of  that  measure  of  L. 

2 

To  the  extent  that  problems  with  the  weapon  system  arise  after 
IOD  and  require  re-engineering  or  re-tooling,  this  end  point  for  pro¬ 
gram  length  will  be  deficient.  This  may  be  a  problem,  for  example, 
for  the  F-lll  and  the  C-5A  programs. 

This  discussion  of  Fig.  7  reveals  another  way  of  interpreting  the 
predictability  of  production  outcomes  from  development  activity  (as 
hypothesized  in  equations  (3)  and  (A)  in  Section  II)  .  The  total  cost 
estimate  at  the  beginning  of  the  program  can  be  broken  down  as  the  sum 
of  estimates  oi  development  and  production  costs  (that  is ,  C  -  C 

eo  0 

+  C  ).  Similarly  an  estimate  cf  the  costs  could  be  made  near  the  end 
P0 

of  development  at  IOD  (C  ■  C  +  C  ) .  At  the  end  of  the 

elOD  iOD  PI0D 

program,  the  actual  costs  are  known  (C  ■  C,  +  C  ).  Since  develop- 

fl  U  U 

a  a 

ment  is  essentially  over  by  IOD,  the  estimate  C  will  be  very  close 

iOD 

to  Cj  .  The  additional  assumption  embodied  in  equations  (3)  and  (A) 
a 

is  that  since  the  configuration  is  resolved  by  the  development  program, 
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inefficient  in  production,  the  total  cost  of  the  weapon  system  could 
be  estimated  with  reasonable  certainty  by  IOD .  In  the  empirical  anal¬ 
ysis,  any  inefficiency  in  production  will  probably  contribute  to  an 
upward  influence  on  the  coefficient  of  this  measure  of  program  length 
or  on  the  intercept,  so  that  although  inefficiency  is  not  measured 
separately,  it  is  not  ignored. ^ 

In  most  of  the  empirical  results  presented  below,  however,  a 
somewhat  different  measure  of  time  in  development  is  actually  used. 

The  new  measure  is  required  since  the  choice  of  program  length  for 
the  "1950s"  is  otherwise  rather  arbitrary.  The  data  for  the  "1950s" 
include  cost  factors  calculated  from  estimates  made  at  various  points 
throughout  the  program  (see  Table  8),  rather  than  one  cost  factor  for 
each  program  calculated  from  an  estimate  made  near  the  time  of  program 
approval.  It  was  not  possible  to  establish  a  beginning  point  for  pro¬ 
gram  length  (and  obtain  an  estimate  made  at  that  time)  for  the  systems 
in  the  "1950s"  sample  comparable  to  that  used  for  the  programs  of  the 
"1960s"  sample.  IOD  was  still  used  as  the  end  point. 

For  the  "1960s"  sample  there  are  also  estimates  available  at 
several  points  in  the  program  which  can  be  used  in  the  analysis.  A 
new  variable  "M"  therefore  has  been  defined  as  the  number  of  months 
between  the  date  of  the  estimate  on  which  each  cost  factor  is  based 
and  IOD.  This  variable  (M)  will  probably  average  fewer  months  than 
would  program  length  (L)  ,  but  hopefully  will  provide  a  better  compari¬ 
son  uhan  would  using  earliest  estimates  or  all  estimates  in  the  "1950s," 
and  comparing  those  results  to  estimates  made  near  the  time  of  approval 
in  the  "1960s." 


C  will  be  quite  close  to  C  .  Any  remaining  bias  in  predicting 
PI0D  Pa 

production  costs  should  be  captured  by  the  intercept  "a"  in  equation 

(11). 

^By  measuring  L  as  the  time  from  program  approval  to  IOD,  we  pre¬ 
clude  any  possibility  of  the  model  evaluating  inefficiency  in  production 
separately.  This  task  is  also  made  difficult,  however,  by  the  use  of 
cost  factors,  which  lump  estimation  inaccuracy  and  inefficiency  together. 
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IV.  COMPARISON  OF  THE  1950S  AND  1960S 


From  the  empirical  model  and  data  measurements  described  in 
Section  III,  the  cost  experience  of  the  last  two  decades  can  be  ana¬ 
lyzed.  This  will  be  done  not  only  by  estimating  the  model  with  the 
full  sample  for  the  "1950s"  and  "1960s,"  but  also  using  two  subsamples . ^ 

The  full  coverage  of  the  "1950s"  sample  includes  fighter,  bomber, 
cargo,  and  tanker  aircraft,  and  ICBM,  IRBM,  and  other  missiles;  the 
"1960s"  sample  includes  fighter,  cargo,  attack,  and  STOL  aircraft  as 
well  as  a  helicopter,  sonars  and  a  sonobuoy,  a  space  propulsion  vehicle, 
a  battlefield  support  missile,  and  two  programs  related  to  an  ICBM. 

The  primary  subsample,  aircraft  and  missiles,  actually  includes  the 
full  sample  for  the  "1950s,"  but  eliminates  the  sonars  and  sonobuoy 
from  the  "1960s"  sample.  Therefore,  this  subsample  provides  the  most 
comparable  results  for  the  comparison  of  acquisition  cost  experience 
between  the  last  two  decades.  The  other  subsample  reported  here  Includes 
only  aircraft. 


EQUATIONS 

We  shall  first  explore  the  appropriate  transformation  of  A  to  use 
as  a  proxy  for  E.  To  accomplish  this,  the  variable  M  defined  at  the 
end  of  Section  III  will  be  used  throughout  as  a  proxy  for  L.  An  addi¬ 
tive  function  of  the  proxies  for  development  program  characteristics 
has  been  chosen  so  that  separate  coefficients  can  be  estimated  for  M 


As  mentioned  in  Section  III,  the  headings  "1950s"  and  "1960s"  are 
used  throughout  for  convenience  of  reference.  Actually  the  "1950s" 
sample  spans  the  time  from  programs  begun  in  1947  to  those  operational 
in  1961,  while  the  "1960s"  programs  were  begun  anywhere  from  1958  to 
1966  and  became  operational  by  1969.  The  "1950s"  data  were  obtained  by 
Rand  researchers  in  the  early  1960s;  the  "1960s"  sample  was  obtained 
early  in  1969,  also  by  Rand  researchers.  In  the  latter  sample,  the 
attempt  was  made  to  obtain  a  cost  estimate  established  near  the  time 
of  DoD  approval.  It  was  considered  preferable  to  maintain  the  "1960s" 
sample  to  include  all  of  the  cost  factors  based  on  such  estimates, 
rather  than  to  combine  these  estimates  with  the  others  in  the  "1950s" 
and  thereby  create  non-uniform  sampling  procedures  within  the  data  for 
one  of  the  periods. 
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and  various  functions  of  A.  Thus  the  structure  for  the  model  resulting 
in  the  estimates  presented  in  Table  3  is  linear  in  M  and  f (A)  : 

log  F  -  log  a  +  bM  +  cf (A)  (13) 

With  M  substituted  for  L  in  equation  (12),  this  function  is  derived 
from  the  assumption  that  equation  (7)  relating  program  length  and 
technological  advance  to  effort  takes  the  form: 

E  -  1  +  .  (14) 

bL 

2  A 

Four  functions  f(A)  are  investigated:  loggA,  A,  A  ,  and  e  .  For 

ease  in  comparing  the  degree  of  fit  of  the  various  functions,  Table 

2 

4  presents  the  different  values  of  R  --  the  proportion  of  the  varia¬ 
tion  of  the  cost  factors  explained  by  the  model.  Each  line  contains 

2 

a  separate  sample  coverage,  so  the  values  of  R  are  directly  com¬ 
parable  only  across  a  line.  In  every  case,  an  increasing  function 

of  A  is  preferable  to  simply  A  or  log  A.  Only  for  the  full  sample 

2  e  2 
for  the  "1960s"  does  the  R  loom  slightly  larger  for  the  A  form 

A 

than  for  the  e  form;  therefore,  the  latter  form  is  taken  to  be  the 
best  functional  form  for  this  linear  structure. 

Returning  to  Table  3,  then,  Part  IV  contains  the  coefficient  esti¬ 
mates  for  the  structure  of  most  interest.  In  this  part,  as  well  as  in 
the  others,  the  primary  point  of  interest  is  that  the  coefficient  "b" 
of  the  program  length  surrogate  declines  between  the  "1950s"  and  "1960s," 
while  the  coefficient  "c"  of  technical  difficulty  increases.  In  every 
case,  "b"  for  the  "1960s"  is  essentially  zero  (usually  slightly  negative) 
and  insignificant . 1  By  contrast,  "c"  is  positive  —  quite  significant 
and  usually  about  4  times  as  large  in  the  "1960s"  as  in  the  "1950s." 

Thus  program  length  has  been  the  main  feature  associated  with  cost 
optimism  in  the  "1950s"  while  technical  difficulty  has  been  the  major 
influence  on  cost  factors  in  the  "1960s." 


These  negative  values  of  "b"  lead  to  the  somewhat  peculiar  impli¬ 
cation  (from  equation  (14)]  that  months  of  development  (L)  and  intensity 
of  effort  (E)  are  positively  related  rather  than  inversely.  Since  the 
negative  values  are  all  quite  insignificant,  the  more  appropriate  con¬ 
clusion  may  be  the  f(A)  alone  is  related  to  E;  this  structure  form  is 
taken  up  below. 


Table  3 


VARIOUS 

TRAMS FORMAT IONS  OF  "A"  IN 

LINEAR 

STRUCTURE  OF  THE 

MODEL 

a 

Coverage 

lc  <  a 

b 

c 

Standard 

Error 

R2 

Sample 

I.  log  F  » 

log  a  + 

bM 

+  c  [log  (A)) 

i. 

Aircraft  A  Missiles 
"1950s" 

-  413 
(-0.5) 

.0128 

(4.6) 

.1195 

(0.4) 

.487 

.36 

55 

"1960s" 

-.733 

(-2.6) 

-.0006 

(-0.3) 

.4657 

(3.5) 

.179 

.42 

21 

Combined 

-.802 

(-1.9) 

.0100 

(4.7) 

.3149 

(-’.8) 

.439 

.:  2 

76 

2. 

Aircraft 

"1950s" 

-.763 

(-1.0) 

.0122 

(3.5) 

.3105 

(1.0) 

.436 

.37 

34 

"1960s" 

-.626 

(-2.2) 

.0025 

(0.9) 

.3945 

(2.9) 

.169 

.55 

13 

Combined 

-.801 

(-1.9) 

.0103 

(3.9) 

.3488 

(1.9) 

.384 

.36 

47 

3. 

Full  sample 
"1960s” 

-.753 

(-2.3) 

-.0008 

(-0.3) 

.4971 

(3.3) 

.219 

.34 

25 

Combined 

-.714 

(-1.8) 

.0099 

(4.7) 

.2877 

(1.7) 

.436 

.31 

80 

II.  log  F  “ 

log  a  + 

bM 

+  cA 

1. 

Aircraft  &  Missiles 
"1950s" 

-.294 

(-0.9) 

.0126 

(4.5) 

.0148 

(0.5) 

.486 

.36 

55 

"1960s" 

-.196 

(-1.5) 

-.0006 

(-0.3) 

.0516 

(4.2) 

.165 

.50 

21 

Combined 

-.415 

(-2.2) 

.0097 

(4.5) 

.0326 

(2.0) 

.436 

.33 

76 

2. 

Aircraft 

"1950s" 

-.349 

(-1.1) 

.0120 

(3.4) 

.0292 

(1.1) 

.435 

.38 

34 

"1960s" 

-.168 

(-1.2) 

.0024 

(1.0) 

.0427 

(3.4) 

.157 

.61 

13 

Combined 

-.365 

(-1.8) 

.0100 

(3.8) 

.0353 

(2.1) 

.382 

.37 

47 

3. 

Full  sample 
"1960s" 

-.171 

(-1.1) 

-.0008 

(-0.4) 

.0541 

(3.7) 

.210 

.39 

25 

Combined 

-.361 

(-2.0) 

.0096 

(4.6) 

.0298 

(1.9) 

.434 

.32 

80 
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Table  3  (conc'd) 


^  a 

Coverage 

log  a 

b 

c 

Standard 

Error 

R2 

Sample 

III.  log  F  - 

log  a  +  bM 

+  cA2 

1.  Aircraft  &  Missiles 
"1950s" 

-.234 

(-1.2) 

.0123 

(4.4) 

.0008 

(0.7) 

.486 

.37 

55 

"1960s" 

.037 

(0.4) 

-.0005 

(-0.3) 

.0026 

(4.8) 

.155 

.56 

21 

Combined 

-.254 

(-2.0) 

.0095 

(4.4) 

.0015 

(2.2) 

434 

.34 

76 

2.  Aircraft 
"1950s" 

-.192 

(-1.0) 

.0119 

(3.4) 

.0013 

(1.1) 

.434 

.38 

34 

"1960s" 

.023 

(0.2) 

.0024 

(1.0) 

.0021 

(3.7) 

.148 

.'j 

13 

Combined 

-.187 

(-1.4) 

.0098 

(3.7) 

.0016 

(2.2) 

.380 

.37 

47 

3.  Full  sample 
"1960s" 

.077 

(0.7) 

-.0008 

(-0.3) 

.0026 

(4.0) 

.205 

.42 

25 

Combined 

-.214 

(-1.8) 

.0094 

(4.4) 

.0014 

(2.1) 

.432 

.32 

80 

IV.  log  F  - 

log  a  +  bM 

+  c[eA] 

1.  Aircraft  &  Missiles 
"1950s" 

-.153 

(-1.3) 

.0112 

(4.2) 

. 34E-07 
(1.8) 

.473 

.40 

55 

"1960s  ' 

.182 

(2.6) 

- . 0001 
(-0.1) 

.  16E-06 
(5.3) 

.145 

.62 

21 

Combined 

-.096 

(-1.0) 

.0088 

(4.2) 

. 46E-07 
(3.0) 

.423 

.37 

76 

2.  Aircraft 
"1950s" 

-.053 

(-0.4) 

.0113 

(3.2) 

.  36E-07 
(1.5) 

.428 

.40 

34 

"1960s" 

.126 

(1.5) 

.0026 

(1.2) 

. 14E-06 
(4.5) 

.132 

.72 

13 

Combined 

-.027 

(-0.3) 

.0093 

(3.5) 

. 47E-07 
(2.4) 

.375 

.39 

47 

3.  Full  sample 
"1960s" 

.249 

(2.5) 

- . 0004 
(-0.2) 

. 15E-06 
(3.3) 

.219 

.34 

25 

Combined 

-.069 

(-0.7) 

.0088 

(4.2) 

.42E-07 

(2.8) 

.423 

.35 

80 

Notes : 

The  t-statistics  are  presented  in  parentheses  below  coefficient  estimates. 

In  part  IV,  the  "c"  coefficients  are  very  small  (since  is  usually  very  large);  the  coefficient 
values  are  listed  in  scientific  notation.  For  example,  the  "1950s"  Aircraft  and  Missiles  "c"  coeffi¬ 
cient  is  .34  x  10  ”  7  "  .000000034. 

aThe  "1950s"  sample  includes  only  aircraft  and  missiles,  so  estimates  for  the  "1950s"  (1)  Aircraft 
and  Missiles  sample  and  (3)  Full  sample  of  each  part  are  the  same. 
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Table  4 


R2  FOR 

VARIOUS 

TRANSFORMATIONS 

OF 

"A"  IN 

LINEAR 

STRUCTURE 

Coverage 

log  A 

A 

A2 

A 

e 

Sample  Siz« 

Aircraft  and 

"1950s" 

Missiles 

.36 

.36 

.37 

.40 

55 

"1960s" 

.42 

.50 

.56 

.62 

21 

Combined 

.32 

.33 

.34 

.37 

76 

Aircraft 

"1950s" 

.37 

.38 

.38 

.40 

34 

"1960s" 

.55 

.61 

.65 

.72 

13 

Combined 

.36 

.37 

.37 

.39 

47 

Full  Sample 

"1960s" 

.34 

.39 

.42 

.34 

25 

Combined 


31 


32  .32 


.35 


80 
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Statistical  measures  of  the  question  of  overall  improvement 
between  the  decades  can  be  handled  in  two  ways:  by  a  test  of  equality 
of  the  coefficients  in  the  equations  for  the  two  decades,,  and  by  a  test 
of  the  accuracy  of  prediction  by  one  decade's  equation  of  the  systems 
represented  in  the  other  decade's  sample.  The  former  is  performed  by 
means  of  an  F  test,  the  latter  with  Theil's  U  statistic.^-  Using  the 
aircraft  and  missiles  sample,  the  hypothesis  that  there  is  no  change 
in  the  coefficients  between  the  two  decades  is  not  rejected  by  the 
usual  statistical  criteria  of  a  5  or  1  percent  significance  level. 
However,  the  hypothesis  would  be  rejected  at  about  the  10  percent  level 
(the  F  statistic  is  2.02  with  3  and  70  degrees  of  freedom). 

The  other  indication  of  very  little  change  in  cost  estimation 
accuracy  or  cost  control  over  the  last  two  decades  comes  from  the 
measure  of  predictive  accuracy  of  this  structure  with  "1950s"  param¬ 
eter  estimates  extrapolated  to  the  "1960s"  data.  The  Theil  U  statis¬ 
tic  (which  ranges  from  0  for  perfect  prediction  of  actual  values  to  1 
for  maximum  inequality  between  predictions  and  actuals)  is  .139.  While 
not  terribly  low,  the  predictions  are  not  uniformly  biased  or  asymmetri¬ 
cal  in  their  prediction  of  the  range  of  actual  cost  factors  for  the 
1960s.2 

A  comparison  of  parameter  estimates  between  the  various  sample 
ranges  reveals  that  the  values  are  rather  insensitive  to  the  coverage 
of  systems.  For  the  "1960s"  equation,  only  the  aircraft  sample  pro¬ 
duces  a  value  of  "b"  for  the  preferred  structure  (part  IV)  that  is 


^See  the  "Statistical  Measures"  section  of  the  Appendix  for  brief 
descriptions . 

2 

As  discussed  in  the  "Statistical  Measures"  section  of  the  Appendix, 
the  U  statistic  is  an  overall  measure  of  predictive  accuracy  of  an 
equation.  The  extent  of  inaccuracy  can  be  broken  down  into  the  pro¬ 
portion  of  inaccuracy  of  prediction  due  to  "unequal  central  tendency" 
(UN),  '  'unequal  variation"  or  asymmetry  of  forecast  over  the  range  of 
actual  values  (U^) ,  and  "imperfect  variations"  or  lack  of  high  positive 
correlations  between  actual  and  predicted  values  (U^) .  If  inaccuracy 
of  prediction  is  present,  the  most  desirable  type  is  the  third  (that 
is,  a  large  proportion  of  U^) .  In  this  test,  U  -  .1391,  and  the  extent 
of  inaccuracy  is  broken  down  as  “  .0212,  «  .0002,  and  =  .9786. 
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somewhat  significant  and  positive.  The  values  of  "c"  remain  the  same 
across  the  samples  in  this  part  of  the  table,  but  the  intercept  for 
the  "19608"  increases  with  the  scope  of  the  sample;  from  the  values 
for  the  intercept  in  log  form  one  can  transform  them  to  average  cost 
factors  (before  development  program  characteristics  are  considered) 
of  1.13  for  aircraft,  1.20  for  aircraft  and  missiles,  and  1,28  for  the 
full  sample. 

One  other  structure  for  the  model  [equation  (12)]  is  also  of 
interest.  In  this  form,  the  program  length  surrogate  and  technologi¬ 
cal  advance  function  enter  multiplicatively : 

log  F  -  log  a  +  bM*  f (A)  .  (15) 

The  assumed  form  of  equation  (7)  for  this  structure  is:^ 

E  -  f (A)  ,  (16) 

A 

where  the  f(A)  to  be  explored  will  be  A  and  e  .  The  results  are 
presented  in  Table  5. 

As  in  the  results  for  the  additivn  structure,  the  most  rapidly 

A 

increasing  function  of  A  —  the  e  surrogate  for  E  —  provides  the 

2 

least  squares  fits  with  highest  "R  s;"  the  fits  are  even  slightly 
better  than  for  the  additive  model.  In  every  case  the  "b"  coefficients 
are  positive  and  significantly  different  from  zero.  The  values  are 
slightly  lower  for  the  "1960s"  than  for  the  "1950s"  in  the  "M*A"  for- 

A 

mulation,  but  about  twice  as  high  as  the  latter  in  the  "M*e  ."  Both 
structures  reveal  Intercepts  higher  for  the  "1960s"  than  for  the  "1950s"; 

A 

for  the  "M*e  "  structure  the  intercepts  are  significantly  different  from 
zero  in  log  form  and  therefore  imply  cost  factors  greater  than  unity 

G 

on  the  average,  even  before  taking  account  of  the  relevant  development 
program  characteristics. 

The  significance  tests  reveal  even  less  difference  between  the 
decades  for  the  multiplicative  forms  of  the  "aircraft  and  missiles" 

JThe  assumption  could  be  stated  slightly  more  generally  as  E  -  cf(A). 
However,  the  effect  of  this  would  merely  be  that  the  coefficient  "b" 
would  contain  the  factor  1/c.  If  "c"  remains  the  same  within  each  of 
the  two  decades,  then  the  comparison  between  the  decades  is  not  affected 
by  this  subf titution. 


MULTIPLICATIVE  STRUCTURE 
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equations  than  they  did  for  the  additive  formulation.  In  no  case  is 

the  hypothesis  of  no  change  in  the  coefficients  between  the  decades 

rejected  at  even  the  25  percent  significance  level. ^  And  the  Theil 

U  statistic,  measuring  the  predictive  accuracy  of  the  "1950s"  equation 

in  the  "'1960s'  aircraft  and  missiles"  sample,  has  a  value  of  only 

.10  in  each  of  the  two  f(A)  formulations;  thus,  the  predictive  accuracy 

is  reasonably  good.  Only  about  2  percent  of  the  inaccuracy  is  due  to 

systematic  forecast  bias  in  the  "M*A"  formulation,  but  80  percent  was 

A  2 

attributable  to  systematic  errors  in  the  "M*e  "  forecasts. 

The  "1960s"  equations  provide  considerably  less  accurate  "pre- 

3 

dictions"  of  the  "1950s"  outcomes.  The  Theil  U  statistics  are  .31 

A  4 

for  the  "M*A"  formulation  and  .63  for  the  "M*e  ." 

In  sum,  none  of  the  structures  explored  indicates  a  significant 
difference  between  the  1950s  and  1960s  in  the  ability  of  the  "system 
acquisition  process"  to  estimate  costs  accurately  or  avoid  actual  cost 
overruns  for  a  given  development  program.  Although  the  results  for 
each  decade  are  insignificantly  different  in  a  statistical  sense,  the 
implication  of  some  of  the  coefficient  estimates  in  each  decade  is  that 
the  process  Itself  tends  to  produce  a  higher  cost  factor  for  a  given 
program  in  the  1960s  than  it  would  have  in  the  earlier  years.  Despite 
this  "deterioration"  of  the  process,  the  outcomes  for  the  "1960s"  have 


The  test  for  changes  ir.  the  coefficients  in  part  I,  item  1, 
leads  to  an  F  statistic  of  .55  with  2  and  72  degrees  of  freedom;  for 
part  II,  item  1,  the  F  statistic  is  .98. 

^The  Theil  U  statistic  for  the  "M*e^"  structure  is  .1020  (with 
U  ■  .1921,  uS  »  .6940, and  ■  .1138).  For  the  "M*A"  structure  the 
Theil  U  statistic  is  .1002  (with  U**  -  .0117,  US  -  .0112,  and  \fi  -  .9  772). 

3 

While  extrapolation  of  an  equation  backward  in  time  is  of  less 
interest  in  a  policymaking  sense,  it  is  a  useful  exercise  in  determining 
whether  the  "system  acquisition  mechanism"  has  undergone  a  net  improve¬ 
ment  between  decades. 

4  A 

The  complete  set  of  statistics  shows  the  "M#e  "  structure  for  the 

"'1960s'  aircraft  and  missiles"  equation  has  a  U  statistic  of  .6349 
[with  uM  ■  .1464  (predictions  exceeding  actual  cost  factors  on  the 
average),  -  .7725,  and  ■  .0811];  for  the  "M*A"  structure  the  U 
statistic  is  .3097  [with  ■  .1186  (predictions  less  than  actuals 
on  the  average),  I'S  ■  .7339  ,  and  ■  .1475]. 
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been  no  worse  than  in  earlier  years  because,  in  part,  the  level  of 
technological  advance  attempted  has  been  kept  lower  on  the  average. 

I  will  take  up  this  aspect  of  the  comparison  between  the  decades  in 
the  "Discussion"  subsection  below. 

Two  further  questions  about  the  equations  should  be  considered. 
First,  how  sensitive  are  the  results  to  the  measure  of  program  leagth 
that  has  been  used;  and  second,  how  sensitive  are  the  results  to  the 
weighting  of  the  programs  within  the  sample? 

As  discussed  in  Section  III,  the  best  available  measure  of  pro¬ 
gram  length  for  the  "1960s"  is  the  number  of  months  from  the  date  of 
the  estimate  established  near  the  time  of  DoD  approval  of  initiation 
of  development  to  the  initial  operational  delivery.  This  amounts  to 
using  only  one  estimate  for  each  system  in  the  sample  —  the  earliest. 
In  Table  6,  the  same  selection  of  observations  from  the  sample  has 
been  used  for  the  "1950s,"  although  these  do  not  necessarily  establish 
the  beginning  of  development;  in  some  cases  they  are  merely  preliminary 
guesswork  at  an  early  stage  of  system  conceptualization.  Therefore, 
this  form  of  the  data  has  not  been  used  for  comparison  of  system 
acquisition  experience  between  the  decades.  Table  6  presents  the 
estimates  using  this  "earliest  M"  for  each  system  as  a  surrogate  for 
L  and  shows  that  by  comparison  with  the  results  in  Table  5,  there  are 

only  quite  minor  differences:  the  intercepts  for  the  "1950s"  are 

2 

usually  somewhat  higher  and  the  "R  s"  lower  in  the  "M*A"  formulation , 
while  the  corresponding  intercepts  for  the  "1950s"  are  lower  in  the 
"M*e  "  formulation.  The  results  of  the  statistical  tests  are  essen¬ 
tially  unchanged.^-  Therefore,  the  results  on  the  whole  appear  to  be 
reasonably  insensitive  to  the  particular  choice  of  a  surrogate  for 
development  program  length. 

Table  7  displays  the  results  of  estimating  the  "1960s"  multipli¬ 
cative  formulation  with  each  system's  observation (s)  weighted  by  the 

"''For  the  "M*A"  structure:  the  F  statistic  is  .55  with  2  and  29 
degrees  of  freedom,  and  the  prediction  measures  for  the  "'1950s'  air¬ 
craft  and  missiles"  equation  in  the  "1960s"  data  are  U  -  .1094  (with 
\P  m  .2097,  ■  .0002,  and  ■  .7901).  For  the  "M'e^"  structure: 

the  F  statistic  is  .35,  and  U  =  .0814  (with  ■  .0671,  =  .7402,  and 

Uc  -  .1927). 


BASED  ON  EARLIEST  ESTIMATE  USED  TO  APPROXIMATE  "L"  IN  MULTIPLICATIVE  STRUCTURE 
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Table  7 

MULTIPLICATIVE  STRUCTURE  FOR  "1960S"  WEIGHTED  BY  PROGRAM  SIZE 


a 

Coverage 

log  a 

b 

Standard 

Error 

R2 

Sample 

I.  log 

F  ■  log  a 

+  bM*A 

i. 

Aircraft  &  Missiles 

.100 

(1.3) 

.00061 

(4.6) 

.172 

.53 

21 

2. 

Aircraft 

.123 

(1.5) 

.00062 

(4.6) 

.143 

.66 

13 

3. 

Full  sample 

.109 

(1.5) 

.00060 

(4.8) 

.177 

.50 

25 

II.  log 

F  -  log 

a  +  b(M*e^) 

1. 

Aircraft  &  Missiles 

.226 

(7.4) 

•32E-08 

(9.5) 

.104 

.83 

21 

2. 

Aircraft 

.244 

(6-9) 

. 31E-08 
(8.3) 

.091 

.86 

13 

3. 

Full  sample 

.234 

(7.4) 

.32E-08 

(8.9) 

.119 

.78 

25 

Notes i 

aSe.e  notes  to  Table  5. 
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size  of  the  program  (measured  as  total  dollars  actually  spent) .  1  lese 

results  indicate  a  slight  upward  shift  of  the  curves  from  those  pre¬ 
sented  in  Table  5  —  the  "b"  coefficient  is  higher  in  the  aircraft  and 
missile  and  full  samples  and  only  slightly  lower  in  the  aircraft 
sample.  The  intercepts  are  higher  for  both  subsamples  but  slightly 
lower  for  the  full  sample.  Overall,  these  results  are  also  little 
different  from  the  main  results  on  the  multiplicative  structure 
presented  in  Table  5. 

Let  us  turn,  then,  to  further  interpretations  of  the  results  of 
Tables  3  and  5,  and  the  implications  of  these  results  for  the  evalu¬ 
ation  of  system  acquisition  experience. 

DISCUSSION 

In  this  subsection  I  present  the  results  displayed  above  in 
briefer  graphic  form.  I  shall  concentrate  on  three  structures  for 
the  model  set  out  in  Section  III,  equation  (11);  namely,  the  results 
for  the  aircraft  and  missiles  samples  for  the  two  decades: 


Aircraft  and 

Missiles 

ture 

"1950s" 

~  "1960s" 

(a) 

F  - 

...  .0112M+.034xl0_6(eA] 

.  o3oe 

F  -  1  20e',0001M+,l6xl0~6te 

(b) 

F  - 

.00087  M *A 

,917e 

F  -  1.09e-00046  M’A 

(c) 

F  - 

,  ,,  .0015xl0~6[M-eA] 

1 1 1  /  6 

Let  me  first  briefly  review  the  conclusions  from  the  statistical  results 
presented  above. 

The  coefficient  estimates  in  the  various  structures  for  the  model 
can  be  described  as  characterizations  of  the  system  acquisition  pro¬ 
cess  in  each  of  the  two  decades.  Average  values  of  the  development 
characteristics  —  "M"  (months  between  date  of  estimate  and  IOD)  being 
used  as  a  surrogate  for  development  program  length  and  "A"  being  a 


measure  of  technological  advance  sought  —  characterize  the  way  in 
which  developments  have  differed  between  the  decades.  The  statistical 
tests  discussed  above  led  to  the  implication  that  there  has  been  no 
significant  difference  in  the  process  between  the  two  decades;  one 
even  finds  that  the  process  seems  to  have  "deteriorated"  for  some  types 
of  development  programs  in  the  sense  that  for  a  given  set  of  develop¬ 
ment  program  characteristics  the  cost  factor  for  the  "1960s"  implied 
by  the  model  would  be  higher  than  for  the  "1950s."'*’  Despite  this 
implication  for  the  process,  the  programs'  outcomes  —  which  can  be 
characterized  as  having  resulted  from  programs  in  the  1960s  that  were 
somewhat  shorter  and  of  somewhat  lower  technological  difficulty  on  the 

average  —  have  shown  a  "typical"  1960s  program  to  have  a  somewhat 

2 

lower  cost  factor  than  a  "typical"  1950s  program.  That  is: 


This  is  a  somewhat  different  conclusion  from  that  presented  by 
Perry  et  al. ,  Systems  Acquisition  Experience,  in  which  we  concluded 
(p.  vi): 

The  analysis  shows  that,  on  average,  cost  estimates  for 
the  1960s  were  about  25  percent  less  optimistic  than  those 
for  programs  of  the  1950s.  Thus,  if  reduction  in  bias  (or 
reduced  optimism)  is  a  realistic  index  of  "better,"  there 
is  evidence  of  improvement  in  the  acquisition  process. 

The  model  on  which  that  analysis  and  the  present  one  were  based  has 
remained  essentially  the  same  (although  the  specific  structural  speci¬ 
fications  for  the  model  have  been  further  refined).  The  main  reason 
for  the  changes  in  the  implications  is  an  improved  and  up-dated  data 
base  for  the  "1960s."  All  the  data  used  in  that  earlier  analysis 
have  been  rechecked  and  a  few  errors  detected  and  eliminated.  But 
mainly,  the  estimate  of  actual  cost  of  the  more  recent  acquisitions 
(especially  C-5A  and  F-lll)  have  been  updated  to  December  1969  esti¬ 
mates.  The  earlier  analysis  used  estimates  of  about  a  year  earlier 
(the  earlier  C-5A  estimated  actual  cost  had  been  made  in  October  1968; 
the  F-lll  had  been  made  March  1969).  See  the  Appendix  for  additional 
discussion. 

2 

There  is  nothing  contradictory  in  this  statement.  In  fact,  it 
has  a  parallel  in  the  observation  that  although  a  system  may  cost  more 
per  unit  than  expected,  the  total  actual  expenditure  may  not  differ 
greatly  from  the  total  anticipated  —  only  the  quantity  procured  may 
have  changed. 

The  average  values  of  M  in  the  table  are  based  on  the  full  sample 
of  systems;  they  are  therefore  probably  low  estimates  of  development 
program  length  (see  Section  III) .  Average  A  is  computed  with  each 
system  weighted  equally. 
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Coverage 

Average  Value 

"Typical 

"  Cost  Factor  for 
Structure 

M 

A 

(a) 

(b) 

(c) 

"1950s" 

42.8 

12.2 

1.40 

1.44 

1.19 

"1960s" 

37.8 

8.9 

1.20 

1.27 

1.20 

"Typical" 

cost  factors 

for  structures  (a 

)  and  (c)  could  be 

derived 

equally  plausibly  from 

the  average  of  the  program  effort  measure  (E, 

which  is 

approximated  by  functions  that 

include  e  ) . 

The 

results  of 

this  exercise  show  a  somewhat 

lover  "typical"  1960s 

cost  factor  for 

structure 

(c)  also: 

"Typical 

"  Cost 

Factor  for 

Average  Value 

Structure 

Coverage 

M 

eA 

(a) 

(c) 

"1950s" 

42.8 

14.6 

e 

1.49 

1.35 

"1960s" 

37.8 

12.3 

e 

1.24 

1.23 

For  consistency  of  presentation,  the  graphic  comparison  of  the  decades 
by  each  of  the  three  structures  (Fig.  9)  will  make  use  of  the  average 
"A"  results. 

To  gain  an  appreciation  for  the  implications  of  these  results 
consider  Fig.  8.  It  is  most  convenient,  in  presenting  a  figure  in 
two  dimensions,  to  consider  first  the  equation  for  each  of  the  three 
structures  of  the  model  that  was  obtained  from  the  aircraft  and 
missiles  sample  for  the  two  decades  combined.  Thus,  each  part  of 
Fig.  8  presents  a  single  equation  plotted  for  4  different  values  of 
A.  For  structures  (a)  and  (c)  —  those  with  the  transformation  of 
technological  advance  sought  (A)  to  effort  per  month  of  development 
(E)  involving  e^  —  the  increases  in  A  have  little  effect  until  A 
gets  to  the  largest  end  of  the  sample  range.  Since  these  are  the 

2 

best  structures  (in  the  statistical  sense  of  having  the  largest  "R  s") , 
the  implication  is  that  seeking  very  large  technological  advancement 
is  the  main  reason  for  serious  underestimation  or  actual  cost 
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overruns.  Since  A  enters  directly  in  structure  (b)  ,  the  curves  for 
equal  changes  in  A  rise  uniformly. 

Let  us  move  on,  then,  to  Fig.  9.  In  this  figure  as  in  the  pre¬ 
vious  one,  a  horizontal  line  at  the  level  of  unity  (cost  factor  =  1) 
would  be  interpreted  as  a  perfect  cost  estimation  or  as  perfect  cost 
control  at  the  level  of  the  cost  estimate  for  any  length  of  time  prior 
to  operational  status  of  the  system.  The  feature  that  has  been  observed 
and  that  this  analysis  has  been  directed  toward  is  that  cost  estimation 
or  inefficiency  typically  leads  to  actual  cost  exceeding  estimated 
cost.  The  extent  of  bias  in  each  of  the  two  decades  is  conveyed  by 
the  height  of  the  curve  above  the  line  at  the  unity  level. 

The  band  around  each  regression  line  corresponds  to  one  standard 
2 

error  of  estimate.  The  shading  range  along  the  horizontal  axis  indi¬ 
cates  each  sample's  range  of  times  of  estimates  (used  in  calculating 
M) .  The  broken  lines  beyond  the  shading  indicate  extrapolations  from 
these  sample  ranges.  The  "1950s"  curves  are  plotted  for  the  average 
value  of  A  for  that  decade  (12.2),  while  the  "1960s"  curves  are  at 
the  level  of  that  decade's  average  value  of  A  (8.9).  The  graphical 
interpretation  of  the  statistical  insignificance  of  the  differences 
between  the  equations  and  of  the  high  predictive  ability  of  the  "1950s" 
equations  for  the  "19603"  data  is  that  the  curves  for  the  "1960s"  falls 
almost  entirely  within  these  one  standard  error  bands  over  the  entire 
range  of  "1960s"  values  of  M.  The  bands  themselves  indicate  a  nar¬ 
rowing  of  the  dispersion  aspect  of  accuracy  in  predicting  costs;  the 
"1960s"  equation  has  a  narrower  one  standard  error  band  for  each 
structure  than  does  the  "1950s"  equation. 


However,  the  results  may  merely  indicate  that  the  survey  measure 
used  for  A  is  not  very  sensitive  to  lower  levels  of  technological  advance, 
but  can  only  distinguish  these  from  the  very  highest.  Resolution  of 
this  question  of  the  quality  of  the  measure  versus  a  real  implication 
about  our  past  experience  must  await  more  refined  measurement  of  A. 

2 

See  the  "Predicting  Cost  Growth  and  Uncertainty"  section  of  the 
Appendix  for  a  discussion  of  the  method  used  to  plot  these  curves  from 
the  equations  in  Tables  3  and  5. 
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Structure  (a)  reveals  that  when  the  separate  effects  of  program 
length  and  technological  advance  on  cost  factors  are  measured,  only 
the  "1950s"  experience  seems  to  have  been  sensitive  to  program  length. 

In  this  part  of  Fig.  9  the  height  of  the  "1950s"  curve  for  a  42.8  month 
development  program  is  1.40,  the  "typical"  cost  factor  for  structure 
(a)  presented  above.  Moving  along  the  "1950s"  curve,  the  height  at 
37.8  months  of  the  same  technological  advance  level  is  1.32.  The 
shift  to  the  "1960s"  curve  and  the  lower  average  level  of  A  leads  us 
to  the  height  of  the  "typical"  cost  factor  for  the  "1960s"  from  struc¬ 
ture  (a),  1.20.  Thus,  the  statistical  result  that  the  system  acquisition 
process  is  essentially  unimproved  over  the  two  decades  is  compatible 
with  the  result  that  cost  factors  of  (different)  "typical"  programs 
have  declined  for  the  last  decade  from  the  preceding. 
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V .  HYPOTHETICAL  PROCUREMENT 


One  of  the  limitations  of  cost  factors  mentioned  in  the  introduc¬ 
tion  is  that  they  give  no  indication  of  the  magnitude  of  program  costs. 
Let  us  make  such  a  translation  by  considering  the  acquisition  of  a  new 
weapon  system  —  a  fighter  aircraft.  We  can  also  consider  the  fol¬ 
lowing  question:  Based  on  past  experience,  how  can  we  anticipate  the 
extent  to  which  the  actual  cost  of  the  proposed  fighter  will  differ 
from  the  estimate?  That  is,  how  can  the  bias  of  the  cost  estimate 
or  the  inefficiency  of  the  procurement  process  be  quantified  to  yield 
a  cost  figure  that  is  likely  to  be  close  to  the  actual  cost?  The 
results  of  Section  IV  could  be  us?d  to  obtain  such  an  approximation 
of  actual  cost.  The  following  example,  however,  illustrates  the 
inherent  uncertainty  of  such  a  process. 

To  apply  the  empirical  results,  we  must  assume  that  experience  in 
the  future  will  be  similar  to  what  we  have  observed  in  the  1960s;  then 
the  equations  estimated  for  the  "1960s"  aircraft  sample  presented  in 
Tables  3  and  5  can  be  used  to  predict  the  possible  bias  of  estimate 
or  cost  overrun  (see  Fig.  10). ^  Suppose  the  new  fighter  is  subjectively 
assessed  to  require  a  technological  advancement  of  12  by  comparison  with 
systems  in  the  "1960s"  sample. 

Then,  if  the  predicted  cost  of  this  new  weapon  system  is  one  bil- 

2 

lion  dollars  and  it  is  expected  to  be  operational  in  42  months,  the 
best  point  estimate  from  structure  (a)  is  that  the  billion-dollar 

3 

program  would  actually  cost  an  additional  $290  million  (F  ■  1.29). 

But  the  range  of  possible  cost  variation  is  at  least  plus  or  minus 

^See  the  "Predicting  Cost  Growth  and  Uncertainty"  section  of  the 
Appendix  for  a  detailed  discussion  of  the  use  of  these  equations  in 
deriving  F  values. 

2 

This  is  the  average  development  program  length  for  systems  in 
the  "1960s"  sample,  based  on  the  lengths  of  time  from  DoD  approval  of 
development  of  each  system  to  operational  status. 

3 

In  this  discussion,  I  explicitly  discuss  only  the  results  in 
part  (a)  of  Fig.  10.  The  other  two  structures  lead  to  best  point 
estimates  of  additional  cost  of  (b)  520  million  and  (c)  270  million. 
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F»g.  10 — Example  of  prediction  uncertainty 


Cost  factor 
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one  standard  error  of  this  '"1960s'  aircraft"  equation.  That  gives 
us  a  range  (represented  by  the  right-hand  side  of  the  box  in  Fig.  10) 
from  $130  million  more  than  the  predicted  cost  (F  a  1.13)  to  $480 
million  more  than  predicted  (F  ■»  1.48). 

Furthermore,  the  fighter  may  not  become  operational  in  the  length 
of  time  predicted;  it  might  actually  remain  in  development  for  an 
extra  6  months.  What  needs  to  be  emphasized  here  is  that  at  the  time 
a  program  is  under  consideration,  we  are  no  more  certain  of  the  exact 
schedule  within  which  it  will  come  to  fruition  than  we  are  of  the 
exact  cost.  In  the  past,  both  of  these  have  generally  been  different 
from  expectations.  The  range  of  uncertainty  then,  assuming  a  schedule 
slippage  of  6  months,  is  portrayed  by  the  left-hand  side  of  the  box. 

At  this  point  (M  -  48)  the  curve  itself  is  higher  and  the  range  of 
uncertainty  is  greater  —  ranging  from  an  additional  $150  million 
(F  ■  1.15)  to  an  additional  $500  million  (F  ■  1.50).^ 

Thus,  when  inaccuracy  in  the  predictions  or  inefficiencies  in  the 
execution  of  both  cost  and  schedule  aspects  of  programs  are  considered, 
the  overall  range  of  uncertainty  is  represented  by  the  diagonal  of  the 
box  in  part  (a)  —  130  million  to  $500  million  more  than  the  original 


The  additive  structure  (a)  differs  in  an  important  respect  from 
structures  (b)  and  (c).  The  model  [in  equation  (11),  Section  III] 
postulates  that  co^t  factors  are  related  to  development  program  length 
(L)  and  intensity  of  effort  (E) .  All  structures  use  M  (months  from 
date  of  estimate  to  initial  operational  delivery)  as  a  surrogate  for 
L.  Structures  (b)  and  (c)  use  a  function  of  A  as  a  surrogate  for  E 
[see  equation  (16),  Section  IV].  But  structure  (a)  uses  an  expression 
involving  A  and  L  as  a  surrogate  for  E  [see  equation  (14),  Section  IV], 
Therefore,  if  the  left-hand  side  of  the  box  in  part  (a)  of  Fig.  10  were 
drawn  on  the  same  curves  as  the  right-hand  side,  the  drawing  would 
merely  depict  a  schedule  stretch-out  (lengthened  ’  but  reduced  E) , 
not  a  schedule  slippage  (lengthened  L  and  E  held  tne  same)  as  in 
parts  (b)  and  (c) .  Therefore,  the  appropriate  left-hand  side  of  the 
box  is  the  range  of  a  new  curve  with  A  increased  sufficiently  for  the 
result  of  this  new  A  and  program  length  of  48  in  equation  (14)  to  be 
equal  to  the  value  of  the  old  A  of  12  and  program  length  of  42  in 
that  equation.  The  required  Increase  in  A  for  this  adjustment  was 
.134  which  produced  the  slightly  higher  set  of  curves  in  part  (a). 
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prediction.  If,  in  addition,  the  scope  of  the  program  were  increased 
(for  example,  a  fighter  with  even  greater  range  was  requested  by  the 
user's  command),  the  A  would  be  larger  in  fact  and  the  whole  box  would 
shift  upward. 

Finally,  the  confidence  that  we  have  in  this  range  of  estimates 
depends  on  the  extent  to  which  we  believe  that  the  influences  affecting 
the  development  of  this  new  system  are  suitably  reflected  in  the  rather 
small  sample  of  the  1960s  systems  analyzed  above.  Even  if  this  new 
program  is  sufficiently  similar  that  we  can  have  some  confidence  in 
this  rough  approximation  of  the  extent  of  our  uncertainty  about  costs 
of  the  new  system,  the  uncertainty  is  rather  large  —  on  the  order  of 
one-third  to  one-half  billion  dollars  for  a  billion-dollar  program. 


H/ithout  carrying  through  the  same  discussion  for  parts  (b)  and 
(c)  of  Fig.  10,  it  should  be  clear  that  the  comparable  range  of  un¬ 
certainty  for  the  multiplicative  structure  using  e^  is  (c) :  $140 

million  to  $420  million  more  than  the  prediction.  The  multiplicative 
structure  with  simple  A  entered  (as  a  surrogate  for  E)  produces  a 
somewhat  higher  range  (b) :  $300  million  to  $850  million  more  than 

predicted. 
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Appendix 

DATA  BASE  AND  SUPPLEMENTARY  PROCEDURES 


DATA  BASE 

The  two  sets  of  data  are  labeled  "1950s"  and  "1960s"  since  the 
weapon  systems  in  each  set  are  predominantly  from  these  periods.  How¬ 
ever,  the  actual  range  of  sample  observations  (from  earliest  date  of 
estimate  to  latest  IOD)  for  the  "1950s"  is  February  1947  to  July  1961, 
and  for  the  "1960s"  January  1958  to  December  1969. 

Sources  of  Data 

The  cost  factors  and  months  between  the  dates  of  estimate  and 
initial  operational  delivery  are  presented  in  Table  8.  The  data  base 
for  the  "1950s"  was  compiled  by  Eugene  Brussell  of  Rand  and  has  been 
used  in  several  past  Rand  studies.  This  sample  includes  only  Air 
Force  systems.  The  cost  factors  are  those  calculated  and  analyzed 
by  Robert  Summers.1  They  are  based  on  unit  costs  of  production  items 
(for  the  "1960s"  they  are  based  on  total  cost,  which  includes  develop¬ 
ment  and  production).  However,  the  development  portion  of  the  total 
cost  of  systems  of  the  1950s  was  relatively  small  except  for  the  ICBMs 
so  that  the  effect  of  this  difference  in  the  data  is  minimal. 

The  "19608"  data  base  originally  consisted  of  the  questionnaire 

returns  for  a  survey  of  21  systems  selected  by  DDR&E  with  the  advice 
2 

of  the  services.  This  sample  includes  Army,  Navy,  and  Air  Force 
programs.  Originally,  sufficient  data  to  compute  cost  factors  were 
available  for  12  of  these  systems.  Additional  data  have  become 

^Summers,  Cost  Estimates  as  Predictors....  Following  Summers, 
three  of  the  "1950s"  observations  were  not  used  in  the  analysis  because 
these  extremely  early  estimates  were  for  systems  drastically  different 
from  the  systems  ultimately  procured.  These  data  are  enclosed  in 
brackets  in  the  listing. 

2 

The  survey  methodology  and  data  base  are  described  in  Perry 
et  al.  ,  System  Acquisition  Experience',  Sections  I  and  II-  The  survey 
results  were  compiled  by  Dominic  DiSalvo  and  James  Stucktr,  who  also 
contributed  to  the  additional  refinement  of  the  sample  undertaken  by 
Susan  Henrichsen  and  reported  below. 


DEFLATED  COST  FACTORS  AND  MONTHS  TO  FIRST  DELIVERY 
("1950*"  and  ,,1960s") 
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avail  ab  le  since  the  completion  of  the  survey  in  the  summer  of  1969; 
the  original  data  base  has  now  been  revised,  up-dated,  and  expanded  to 
include  a  total  of  25  observations  for  15  systems.  The  revision  has 
included  a  review  of  the  cost  data  available  for  all  systems  in  the 
sample.  Several  sources  were  used  to  update  and  verify  the  s arvey 
data  and  to  obtain  new  information  from  which  additional  cost  factors 
could  be  calculated.  The  sample  was  also  expanded  by  the  inclusion  of 
two  additional  systems,  A-7D  and  A-7E.  The  SQS-26  program  was  separated 
into  two  systems,  SQS-26AX  and  SQS-26CX.^  "Actual"  costs  for  six  of 
the  systems  (F-lll,  C-5A,  A-7D,  A-7E,  Difar,  and  SQS-26CX)  are  in 
reality  the  most  recent  projections  of  what  the  actual  costs  will  be; 
these  programs  are  all  still  in  production  and  actual  costs  are  there¬ 
fore  not  yet  known  with  great  accuracy. 

Of  the  total  amount  of  program  costs  represented  by  this  sample, 
80.3  percent  is  for  Air  Force  systems,  12.2  percent  for  Navy  systems, 
and  7.5  percent  for  Army  systems.  Approximately  half  of  the  sample 
(8  systems)  accounts  for  93.6  percent  of  the  total  cost,  while  two 
very  large  programs  (C-5A  and  F-lll)  account  for  52.7  percent  of  the 
total. 

The  figures  listed  in  the  "Months"  column  of  Table  8  give  the 
number  of  months  from  the  date  of  the  estimate  to  IOD.  In  some  cases, 
there  were  two  dates  associated  with  one  estimate:  the  date  the  esti¬ 
mate  was  "established"  and  the  date  of  the  document  in  which  the  esti¬ 
mate  appeared.  Whenever  this  occurred,  the  "date  established"  was 
used.  It  was  possible  to  determine  IOD  dates  rather  firmly  for  all 
systems  but  one.  No  IOD  date  was  given  for  SQS-26AX,  and  it  was 
necessary  to  approximate  this  date  from  other  information.  The  figure 
given  in  Table  8  is  known  to  be  correct  within  four  months,  more  or 
less . 


The  total  SQS-26  program  extends  over  a  very  long  period  of  time 
(more  than  10  years)  and  has  undergone  substantial  change.  In  addition, 
the  AX  version  and  the  CX  version  were  estimated,  contracted  for,  and 
procured  separately.  The  SQS-26CX  is  currently  being  reported  by  the 
Navy  as  an  individual  weapon  system. 
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Some  uncertainty  exists  for  many  of  the  cost  factors  used  in  the 
analysis,  from  difficulties  in  making  the  necessary  quantity  and 
price-level  adjustments. 

Quantity  Adjustment 

Whenever  quantity  changes  occur  during  a  program,  adjustment 
must  be  made  to  reconcile  the  estimate  with  the  actual  quantity  pro¬ 
duced.  Adjusting  the  estimated  cost  so  that  it  is  for  the  same  quantity 
as  the  actual  cost  is  obviously  preferable  to  calculating  a  cost  factor 
as  a  ratio  of  "total"  costs  for  two  different  quantities.  Such  a  cost 
factor  certainly  would  not  accurately  reflect  the  closeness  of  actual 
to  estimated  cost.  The  accuracy  of  assessment  of  the  cost-quantity 
relationship  for  a  system  is  a  part  of  the  overall  accuracy  of  cost 
estimation.  Therefore,  the  ideal  method  of  quantity  adjustment  is  to 
base  quantity-adjusted  cost  factors  on  estimates  for  the  actual  quantity 
delivered  that  were  generated  by  the  original  estimator  and  used  by 
decisionmakers.  Such  a  cost  factor  would  be  a  useful  ratio  of  the 
actual  to  the  estimated  cost  for  evaluating  estimation  accuracy.  If, 
however,  no  information  is  available  to  indicate  what  the  estimator 
assumed  the  cost-quantity  relationship  to  be,  any  adjustment  introduces 
some  uncertainty  into  the  cost  factor  since  the  estimated  cost  used  to 
calculate  the  factor  has  been  adjusted  by  someone  other  than  the 
estimator . 

For  several  "1960s"  systems  (A-7D,  A-7E,  Pershing  IA,  Minuteman  II  - 
Airborne  Command  Post,  and  Difar) ,  quantity-adjusted  estimates  were 
available;  these  were  used  to  calculate  the  cost  factors.  For  another 
system,  C-141,  the  quantity  change  was  small  enough  that  the  choice  of 
learning  curve  makes  little  difference  in  the  adjusted  cost  factor. 

Two  systems,  C-5A  and  F-lll,  have  undergone  rather  substantial  quantity 
changes,  and  it  was  not  possible  to  determine  with  any  certainty  what 
the  relevant  learning  curves  are.  The  estimates  for  both  systems  were 
adjusted  by  use  of  an  85  percent  learning  curve. ^  Because  the 


I  am  indebted  to  Gilbert  Levenson  for  his  advice  and  assistance 
on  the  necessary  qunatity  adjustments. 
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adjustments  were  rather  large,  considerable  uncertainty  exists  for  these 
cost  factors. 

For  the  "1950s"  systems,  a  review  of  Summers'  underlying  docu¬ 
mentation  for  the  data  reveals  that  quantity  adjustments  were  made 
for  approximately  two-thirds  of  the  cost  factors.  Some  uncertainty 
therefore  exists  for  many  of  these  factors,  because  of  the  difficulty 
of  determining  the  relevant  learning  curve. 

Price  Adjustment 

Some  uncertainty  exists  for  all  cost  factors  owing  to  price-level 
adjustments.  For  purposes  of  this  study,  it  seemed  necessary  to  make 
such  adjustments  for  several  reasons.  First,  it  is  assumed  that  cost 
estimation  accuracy  does  not  include  the  ability  to  predict  price- 
level  changes  accurately.  A  comparison  of  the  acquisition  experience 
of  the  1950s  and  the  19&0s  would  be  biased  by  a  failure  to  adjust  for 
possible  differences  in  the  rate  of  inflation  for  the  two  decades,  over 
which  the  Defense  Department  has  very  little  control. 

Cost  factors  could  be  deflated  accurately  and  with  little  uncer¬ 
tainty  only  if  a  number  of  conditions  were  met.  First,  we  would  have 
to  know  what  allowance  for  inflation,  if  any,  has  been  included  in  the 
estimate  so  that  the  actual  cost  could  be  adjusted  by  the  difference 
between  the  allowance  made  for  inflation  and  the  inflation  that  actually 
occurred.  We  would  also  have  to  know  the  timing  of  program  expenditure, 
and  we  would  need  a  price  index  relevant  to  each  program  in  the  sample. 
At  least  this  much  information  would  be  needed  to  adjust  a  cost  factor 
by  the  difference  between  the  estimated  and  the  actual  rate  of  price 
increase.  Since  such  information  was  not  available  for  any  of  the 
systems,  it  was  necessary  to  devise  some  other  method  of  deflating 
the  cost  factors. 

Several  methods  were  considered  and  regression  results  were 
obtained  for  each  method  in  order  to  ascertain  the  sensitivity  of  the 
results  to  the  method  of  deflation.  These  methods  and  the  results 
will  be  described  below.  First,  regression  results  were  also  obtained 
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for  the  "1950s"  and  the  "1960s"  using  undeflated  cost  factors  to  deter¬ 
mine  to  what  extent  the  magnitude  and  fit  of  the  estimated  coefficients 
depend  on  deflating  cost  factors.  The  results  for  the  two  multiplicative 
structures  for  the  aircraft  and  missiles  sample  (using  undeflated  cost 
factors)  are  shown  below.  The  t  statistics  appear  in  parentheses  below 
the  coefficient  values. 
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Comparison  of  these  results  with  those  shown  in  Table  5  (where  deflated 
cost  factors  were  used)  reveals  that  the  fits  do  not  vary  a  great  deal. 
This  indicates  that  the  explanatory  power  of  the  model  is  not  due  merely 
to  the  use  of  deflated  cost  factors. 


The  "1950s"  cost  factors  were  deflated  by  Summers;  his  method  is 

not  reported,  but  the  resulting  factors  have  been  taken  as  given.  For 

the  "1960s"  factors,  two  different  methods  and  two  different  price 

indexes  were  considered.  The  first  method  (Method  1)  was  to  multiply 

the  undeflated  factor  by  a  figure  equal  to  the  price  index  for  the  year 

in  which  the  estimate  was  made  divided  by  the  price  index  for  the  year 

in  which  IOD  occurred.  In  other  words,  F,  _  ,  “  F  ,  x 

deflated  undeflated 

(P  r  .  /P  ..  ___) ,  This  method  would  be  correct  onlv 

year  of  estimate  year  of  IOD 

if  both  program  expenditure  and  price  increase  were  at  a  constant  rate 
throughout  the  program  and  if  the  time  from  the  date  of  the  estimate 
to  IOD  constituted  half  the  full  program  length.  In  fact,  this  deflation 
is  probably  conservative  both  because  half  the  spending  probably  has 
not  been  completed  by  IOD  and  because  prices  have  generally  increased 
at  an  increasing  rate  in  the  1960s. 


The  second  method  (Method  2)  was  an  attempt  to  deflate  the  "1960s" 
cost  factors  by  the  difference  between  an  approximation  of  the  amount 
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of  price  increase  assumed  by  the  estimator  and  the  actual  amount  of 
price  increase  which  occurred  from  the  date  of  estimate  to  10D.  No 
information  was  available  on  the  amount  of  price  increase  allowed  for 
in  the  estimate.  It  was  therefore  assumed  that  the  estimate  included 
an  allowance  for  a  constant  annual  rate  of  price  increase  equal  to 
the  average  annual  rate  of  increase  for  the  three  years  prior  to  the 
year  in  which  the  estimate  was  made.  This  .resulting  net  price  increase 
was  obtained  by  subtracting  the  "estimator's  assumed  rate"  from  the 
actual  average  rate  of  price  increase  and  deflating  the  cost  factor 
over  the  period  from  the  date  of  estimate  to  IOD.^  In  other  words, 

F 


F  ,  x  [1/(1  +  net  price  increase)  ^ear 

undeflated 


deflated 

year  of  estimate),  ,  ,, 

J.  Since  this  method  assumes  that  some  allowance 

for  inflation  was  included  in  the  estimate  when  in  fact  th^re  may 

have  been  none,  this  method  is  probably  more  conservative  than  Method 

1,  described  above. 


Both  Method  1  and  Method  2  were  used  with  two  price  indexes, 
resulting  in  four  sets  of  deflated  cost  factors.  Using  these  four 
sets  of  data,  the  following  results  were  obtained: 


Deflated  by 

"1960s"  Aircraft 

and  Missiles 

Sample 

Method 

Price 

Index 

log  F  ■  log  a  +  bMA 

log  F  -  log  a  +  bM*e^ 

log  a 

b 

log  a 
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1 
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.086 

.00046 

.179 

.36E-08 

(1.1) 
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(5.9) 

(6.5) 

2 

1 
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.00046 

.198 

. 35E-08 

(1.4) 

(2.7) 

(6.7) 

(6.6) 

1 

2 

.056 

.00023 

.068 

. 33E-08 

(0.6) 

(1.1) 

(1.8) 

(4.8) 

2 

2 

.098 

.00052 

.209 

. 36E-08 

(1.3) 

(3.1) 

(7.6) 

(7.4) 

If  this  method  produced  a  "net  price  increase"  that  was  nega¬ 
tive,  no  further  deflation  was  made  (that  is,  the  "net  price  increase" 
was  set  equal  to  zero) . 

2 

Price  index  1  is  the  Wholesale  Price  Index  for  Machinery  and 
Equipment  available  in  the  1970  Annual  Report  of  the  Council  of  Economic 
Advisors.  Price  Index  2  is  an  unpublished  aircraft  price  index  (ex¬ 
cluding  avionics  and  engines),  developed  by  Harry  G.  Campbell. 
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The  comparison  shown  above  reveals  that  the  regression  results  are 
rather  insensitive  both  to  the  method  of  deflation  and  the  price  index 
used.  The  results  are  nearly  the  same  using  both  methods  with  Price 
Index  1.  When  Method  2  is  used,  the  results  for  both  price  indexes 
are  similar.  The  only  case  in  which  the  coefficient  values  differ 
very  much  is  that  of  using  Method  1  with  Price  Index  2. 

All  of  the  results  reported  in  this  study  used  Method  1  and  Price 
Index  1  (the  Wholesale  Price  Index  for  Machinery  and  Equipment)  ,  to 
deflate  the  cost  factors  of  the  "1960s."  This  index  was  chosen  because 
the  "1960s"  sample  represents  a  number  of  widely  differing  systems  and 
subsystems,  so  that  this  rather  general  index  might  be  more  applicable 
than  would  a  more  specialized  one.  Also,  it  leads  to  relatively  "con¬ 
servative"  deflation  in  that  the  average  rate  of  price  increase  is 
lower  for  it  than  for  Price  Index  2.  In  any  case,  the  choice  of  method 
and  price  fndex  for  deflation  of  the  cost  factors  appears  to  have  a 
minimal  effect  on  the  results. 

Technological  Advance  Sought 

Table  9  displays  the  measures  of  "technological  advance  sought" 

(A)  and  the  contractor  for  each  of  the  systems  in  the  samples.  The 
values  for  A  were  obtained  by  rather  limited  surveys  conducted  within 
Rsr.d  (see  the  discussion  in  Section  III).  These  measures  are,  at 
best,  crude  approximations  of  the  concept  of  the  relative  level  of 
technological  advance  sought  and  should  be  considered  preliminary 
measures  for  this  variable. 

STATISTICAL  MEASURES 

Standard  significance  tests  are  not  really  appropriate  to  the 
kinds  of  data  samples  used  in  thiB  study.  As  discussed  above,  the 
criterion  for  inclusion  of  a  system  in  the  sample  was  the  availability 
of  data:  through  past  Rand  research  in  the  case  of  the  "1950s"  sample, 
and  through  selection  of  systems  by  DDR&E  with  substitutions  made  by 
the  individual  services  for  some  of  the  original  requests  for  the 
"1960s"  sample.  Therefore,  neither  of  them  is  a  random  sample  of  all 


WEAPON  SYSTEMS  AND  "A"  LISTJD  BY  CONTRACTOR 


^-7E  -  LTV 

^-7D  -  LTV 


-63- 


acquisitions  undertaken  during  a  decade.  Still,  significance  tests 
are  commonly  used  to  provide  "feel"  for  the  extent  to  which  the  impli¬ 
cations  of  an  analysis  warrant  acceptance.  For  this  purpose,  the 
following  outline  of  the  statistical  measures  reported  for  the  addi¬ 
tive  and  multiplicative  structures  for  the  model  may  be  helpful. 


The  first  indication  of  significance  is  provided  by  the  value 
of  the  t  statistic  reported  beneath  each  coefficient  in  the  tables. 
For  the  larger  sample  sizes  (over  30)  ,  the  coefficients  are  deemed 
different  from  zero  at  the  10  percent  significance  level  if  t  >_  1.31, 
at  the  5  percent  level  if  t  >_  1.70,  and  at  the  1  percent  level  if 
t  >_  2.46.  For  the  smallest  sample  (7  observations  in  Table  6  and 
thus  5  degrees  of  freedom),  "significant"  t  values  must  exceed  1.48 
(10  percent),  2.02  (5  percent),  and  3.36  (1  percent). 


The  test  of  significantly  different  equations  between  the  decades 
is  accomplished  by  a  test  based  on  the  F  distribution . b  The  multi¬ 
plicative  structure  for  the  model,  for  example,  can  be  written  as 


F 

F 


a50se 


b50sM-f(A) 


a60se 


b60sM‘f(A) 


(17) 

(18) 


and  for  the  estimates  from  an  equation  for  the  two  decades  combined, 
as 

b  M-f(A) 

F  -  a  e  C  .  (19) 

c 

The  test  for  the  entire  equation,  then,  is  based  on  the  hypothesis: 
a50s  “  a60s  and  b50s  “  b60s  ’ 

[in  which  case  equation  (19)  is  sufficient  to  describe  the  past  two 
decades],  with  formulations  (17)  and  (18)  as  the  alternative.  For 


On  such  statistical  tests,  see,  for  example,  J.  Johnston,  Econo¬ 
metric  Methods,  McGraw-Hill  Book  Company,  New  York,  1963,  Sections  4-3 
and  4-4. 
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the  aircraft  and  missiles  sample,  the  significance  levels  are  Fs  of 
slightly  less  then  2.18  [10  percent  level  for  3  and  60  (instead  of  70) 
degrees  of  freedom],  and  2.35  [10  percent  level  for  2  and  60  (instead 
of  72)  degrees  of  freedom]  ,  which  are  almost  never  attained  for  the 
aircraft  and  missiles  results  that  have  been  investigated. 

The  final  statistic  reported  in  Section  IV  is  the  Theil  U  statis¬ 
tic,  which  Theil  calls  an  "inequality  coefficient."^  It  is  calculated 
from  the  actual  values  of  the  cost  factors  (F)  over  the  relevant 
sample,  and  the  predicted  values  (F  )  from  the  equation  under  inves¬ 
tigation;  namely: 


j  -  l  (F  - 
!  n  L  p 

F)2 

V 

n  L  p  f 

"  I  F2 

n  L 

(20) 


Theil  describes  this  statistic  as  follows  (A's  for  actuals,  P's  for 
2 

predicted)  : 

The  coefficient  U  is  —  except  for  the  trivial  case 
where  all  P's  and  A's  are  zero,  when  it  is  indeterminate  — 
confined  to  the  closed  interval  between  zero  and  unity.... 

We  have  U  -  0  in  the  case  of  equality:  P^  ■  for  all  i. 

This  is  clearly  the  case  of  perfect  forecasts.  We  have 
U  ■  1  (the  "maximum  of  inequality")  if  there  is  either  a 
negative  proportionality,  or  if  one  of  the  variables  is 
identically  zero:  rP^  +  sA^  ■  0  for  all  i  and  for  some 
nonnegative  r  and  s  (not  both  zero).  In  other  words, 

U  ■  1  if  there  is  a  non-positive  proportionality  between 
the  P's  and  the  A's.  This  is  indeed  a  case  of  very  bad 
forecasting,  for  it  means  either  that  always  zero  pre¬ 
dictions  of  non-zero  actual  values  are  made  (s  ■  0) ,  or 
that  non-zero  predictions  are  made  of  actual  values 
which  are  always  zero  and  hence  easy  to  predict  (r  ■  0) , 
or  that  predictions  are  positive  (negative)  if  actual 
outcomes  are  negative  (positive)  in  a  remarkably  regular 
manner  (r  and  s  +  0) . 

The  three  types  of  predictive  inaccuracy  are  simply  proportions  of 

the  numerator  of  U  related  to  the  difference  between  the  means  F  and 

P 


^See  H.  Theil,  Economic  Forecasts  and  Policy,  Second  Revised 
Edition,  Amsterdam,  North-Holland  Publishing  Co.,  1961  (Section  2.5). 

2Ibid. ,  pp.  32-33. 
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F,  the  difference  between  the  standard  deviation1!  s  and  s  ,  and  a 

r  £ 

p 

residual  component.  Thus,  the  "unequal  central  tendency"  component  is 


M  "  ^)2 

UM  ,  — £ - 


-l  -  F) 
n  L  p 


2 


(21) 


the  "unequal  variation"  or  asymmetry  of  forecast  component  is 

<SF  ‘  SF)2 
US  -  " 


-  I  (F  -  F) 
n  L  p 
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(22) 


and  the  "imperfect  covariation"  or  residual  component  is 


uc  = 


2d  -  r)sp  sF 


1 

n 


l  (FP  -  F>' 


(23) 


where  r  is  the  correlation  coefficient  between  the  actual  and  pre¬ 
dicted  values.^ 


A  METHODOLOGY  FOR  EVALUATING  CONTRACTORS? 

Table  10  displays  the  results  for  a  variation  of  the  multipli¬ 
cative  structure  that  might  give  us  some  insight  into  the  relative 
"performances"  of  some  of  the  contractors  in  the  defense  industry. 

By  knowing  which  contractor  was  responsible  for  which  weapon  system, 
one  could  hope  to  distinguish  the  "b"  coefficient  for  each  corporation. 
The  individual  corporations  shall  remain  nameless  since  the  results 
of  the  table  do  not  necessarily  indicate  that  a  corporation  is  more 
or  less  efficient  in  developing  weapon  systems.  Although  this  methodo¬ 
logy  is  a  step  toward  evaluating  contractors,  the  data  are  insufficient 
to  the  task  of  distinguishing  relative  corporate  performance. 

"''For  further  details,  see  ibid .  ,  section  2.5.2. 

2 

The  method  of  determining  which  corporations  should  be  included 
in  this  analysis  was  based  solely  on  the  availability  of  the  data 
and  on  the  premise  that  a  corporation  could  be  included  only  if  it 
had  at  least  one  system  in  the  sample  in  each  of  the  two  decades. 


ANALYSIS  OF  CORPORATIONS  USING  MULTIPLICATIVE  STRUCTURE  AND  AIRCRAFT  AND  MISSILES  SAMPLES 


66- 


OJ 

iH 

cx 

LO 

rH 

vO 

m 

rH 

vO 

in 

CM 

UO 

CM 

CO 

<N 

rH 

00 

vO 

o 

& 

m 

00 

<* 

vO 

00 

vO 

• 

• 

• 

* 

• 

• 

T3 

U 

Vi 

CTJ 

0 

o 

rH 

o 

vO 

CO 

rH 

Vi 

in 

rH 

o 

00 

o 

m 

c 

Vi 

rH 

<* 

ro 

rH 

ro 

w 

• 

• 

• 

• 

• 

• 

u 

CO 

co 

/*n 

00  /-N 

m  /'“s 

o 

VO 

00  o 

i 

1  vO 

1 

CO 

rH 

• 

o  • 

o  • 

w 

• 

M  • 

W 

• 

X 

O 

rH 

O  rH 

O  rH 

o 

rH 

n-*  rH 

CM 

rH 

O 

v_/ 

o  ^ 

O 

vf 

vO  w 

ro 

• 

# 

# 

■ 

• 

<r 

vO 

l-- 

VO  /""N 

r>- 

<r\ 

CM 

v£> 

rH  IH 

o  m 

1 

m 

1  <T 

1 

VO 

vO 

rH 

• 

O  • 

rH  • 

w 

• 

W  • 

w 

• 

X 

O 

co 

O  O 

O  ro 

m 

ro 

vO  O 

rH 

CO 

o 

V— «/ 

O  1 

O  w 

rH 

00  1 

rH 

v-/ 

• 

• 

1 

• 

• 

• 

1 

• 

00 

00 

00 

oo 

/— s 

/~s 

CM 

ro 

rH  O 

rH  in 

l 

rH 

1  00 

i 

CO 

in 

o 

• 

O  • 

o  • 

w 

a 

u  • 

w 

• 

X 

o 

O 

O  rH 

o  o 

vO 

o 

vO  O 

m 

o 

o 

N— «' 

O  1 

o  v-*/ 

f*> 

1 

m  l 

rH 

1 

• 

. 

i 

• 

• 

1 

w 

•  ^ 
l 

r 

V 

X 

/"N 

00  ^ 

vO 

On 

/-s 

r-'-  /-~s 

ON 

vO 

in 

1 

vO 

1  crv 

i 

vO 

o 

• 

o  • 

o  • 

w 

• 

w  • 

w 

• 

X 

o 

CM 

O  tH 

o  CM 

<* 

CM 

O  CM 

CM 

CM 

o 

O  ^ 

o  w 

ON 

w 

ON  w 

00 

• 

• 

• 

• 

• 

• 

rH 

fO 

CM  /'-n 

vO 

in 

ON 

ro 

ro 

rH  OV 

1 

m 

1  rH 

1 

CO 

m 

m 

• 

o  • 

rH  • 

w 

• 

W  • 

w 

• 

X 

o 

CM 

O  rH 

O  rH 

CM 

vO  CM 

<r 

rH 

o 

V-h' 

O  w 

O  ^ 

fH 

rH 

• 

• 

* 

• 

• 

00 

/— *s 

00 

CT>  ✓-v 

00 

00  /-N 

00 

00 

o 

vO  vO 

vo 

1 

00 

1  00 

1 

m 

CM 

o 

• 

O  • 

o  • 

w 

• 

w  • 

w 

• 

X 

o 

m 

o  m 

O  m 

n 

vX> 

ov 

CM 

h- 

c 

o  ^ 

o  s-' 

CM 

ro  ^ 

CM 

V 

• 

vD  /*n 

vO 

00 

vO  /-% 

00 

/^s 

i— 1 

o 

O 

O  rH 

co  m 

1 

1  CM 

1 

rH 

X 

rH 

• 

O  • 

O  • 

w 

• 

W  • 

w 

• 

O 

vO 

o  o 

O  vD 

r- 

vO 

r-  O 

vO 

O 

o  \ 

O  ^ 

fH 

rH 

w 

• 

•  V 

1 

* 

* 

• 

\ 

/ - - 

rt 

ro 

*n  ro 

o  in 

r~- 

/*s 

CM 

vO 

n 

• 

uo  * 

■j-  • 

CM 

<1* 

ro  cm 

O 

O 

60 

rH 

rH 

fH  CM 

o  o 

O 

• 

rH  • 

rH 

• 

0 

• 

1 

• 

•  1 

0) 

o 

•  ro 

• 

CM 

iH 

0)  1 

v-/ 

1 

Vi 

3 

3 

4J 

VI 

a 

<J 

3 

3 

OJ 

Vi 

4J 

60 

4J 

"O 

CO 

n 

c/3  r 

z 

0) 

z 

z 

01 

Vi 

m 

00 

c 

I 

a 

(0 

e 

oi 

:  O 

O 

•H 

*< 

o 

o 

•H 

> 

<  m 

vD 

0) 

m 

vD 

o 

•  CTv 

o\ 

B 

• 

ON 

Cv 

CJ 

f  r4 

rH 

0 

u 

f 

iH 

rH 

0 

o 

• 

• 

M 

M 

M 

Vi 

HI 

x 

4-1 

o 

c 


words,  using  the  multiplicative  structure,  a  separate  "b"  coefficient  was  estimated  for  each  corporation  shown 
above,  with  that  corporation's  systems'  program  lengths  and  level  of  technological  advance  used  for  M^  and  A^  . 
The  first  6  numbers  stand  for  six  different  corporations;  the  7th  includes  the  data  for  all  others  that  could 
not  be  separately  estimated.  The  numbers  shown  in  parentheses  below  the  coefficient  estimates  are  t  statistics 
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The  estimate  of  the  "b"  coefficient  merely  describes  the  degree 
to  which  the  contractor's  cost  factors  deviate  from  unity;  it  does  not 
describe  why  the  factors  differ.  Recall  a  point  made  at  the  beginning: 
that  a  cost  factor  cannot  distinguish  between  contractor  inefficiency, 
which  might  lead  to  too  high  an  actual  cost,  and  an  inaccurate  cost 
estimate  made  early  in  the  program.  The  results  appearing  in  Table 
10  might  occur  either  because  cost  estimates  for  systems  developed 
by  certain  contractors  are  uniformly  more  optimistic  than  for  others, 
or  because  of  contractor  inefficiency. 

One  would  thus  want  to  incorporate  a  system  for  evaluating  cost 
estimates  before  he  could  distinguish  contractor  efficiency.  But  even 
then,  the  contractor  could  be  evaluated  only  in  terms  of  cost  per¬ 
formance,  given  the  technological  advance  required.  One  would  want 
to  include  total  system  performance  in  a  full  evaluation  of  a 
contractor. 


PREDICTING  COST  GROWTH  AND  UNCERTAINTY 

In  Section  V,  various  structures  of  the  model  were  used  to  predict 
the  probable  cost  growth  and  range  of  uncertainty  for  a  hypothetical 
procurement.  This  is  done  by  specifying  values  for  the  Independent 
[M  and  f (A) ]  and  calculating  the  values  of  F  from 


F  .  ell°g  a  +  bM  +  cf (A) ] 

for  the  additive  structure,  and  from 
F  _  e(log  a  +  bM* f (A) ] 


(24) 


(25) 


for  the  multiplicative  structure.  The  range  of  uncertainty  considered 
In  Section  V  was  plus  or  minus  one  standard  error  (s.e.).  This  is 
calculated  by  solving 


„  ,  [log  a  +  bM*f(A)  ±  s  e.] 

F  ±  s .e .  ■  e 


(26) 


for  the  multiplicative  structure,  and  similarly  for  the  additive 
structure.  The  coefficients  and  the  standard  error  for  each  equation 
are  given  in  the  tables  in  Section  IV. 


